Watch the video from 'The View'
The women objected to the Fox News Channel host saying that "Muslims killed us on 9/11." They returned after an O'Reilly apology.
The fireworks came amid a heated argument on the issue.
"Listen to me," O'Reilly said, "because you'll learn something."
"Pinhead!" retorted Behar, who also hosts a prime-time show on HLN.
O'Reilly, who was on the popular daytime talk show to promote his book "Pinheads and Patriots: Where You Stand in the Age of Obama," said locating the Islamic Center near ground zero is inappropriate "because Muslims killed us on 9/11."
Goldberg responded with an expletive and Behar rose from her seat.
"I don't want to sit here," Behar said. "I don't. I'm outraged by that statement."
She walked off the set, followed by Goldberg.
The show's creator, Barbara Walters, immediately said her colleagues were wrong to walk off.
"We should be able to have discussions without washing our hands and screaming and walking offstage," she said. But she also scolded O'Reilly, saying that it was extremists who committed the terrorist act. "You cannot take a whole religion and demean them," she said.
"I'm not demeaning anybody," O'Reilly said.
"Yes," Walters replied. "You are."
After some more back-and-forth, O'Reilly said that "if anybody felt that I was demeaning all Muslims, I apologize."
Behar and Goldberg returned, with Behar saying, "We're back now because you apologized."
The 16-story Islamic center and mosque is planned for lower Manhattan, two blocks north of where the World Trade Center once stood. Critics say the location denigrates the memory of those who died on Sept. 11, 2001. Proponents say planners have a constitutional right to build and see the project as a reflection of religious freedom and diversity.
