The 28-year-old model-actress stars in the reality series with Kendra Wilkinson and Bridget Marquardt as one of the 82-year-old publisher's girlfriends who live with Hefner in the Playboy Mansion.
"If Holly says it's over, I guess it's over," Hefner said in a telephone interview Wednesday. "She's still here in the house. Until a few days ago, we were still sharing the same bed."
Hefner was surprised to hear Madison discussing their breakup, but acknowledged he knew a split was imminent after he told Madison that they would never wed or have children.
In a video posted on TMZ.com Tuesday, Madison said she's no longer with Hefner. She also said she is "still filming stuff together" with Wilkinson and Marquardt.
Hefner said Madison learned the pair would never have children or get married six months ago, adding: "The fact that she was depressed after that, I didn't know at all. That was a revelation in the last days and weeks. Quite frankly, we thought when the time came, we would make a combined statement and we expected that combined statement would be somewhere in the weeks and months ahead."
"The Girls Next Door" premiered on E! in 2001 and is in its fifth season. Hefner said he and the three women are committed to a sixth season, but he plans to seek out new live-in lovers. Hefner said 19-year-old Playmate twins Karissa and Kristina Shannon are living in the mansion, but they're aren't his girlfriends - yet.
"It's now apparent there will be some new faces in my personal life and on the show," he said. "There's been moments that I've been down in the dumps about all this, and (personal assistant) Mary (O'Connor) told me to cheer up and pointed out that there are girls lined up outside the front gate. At my age, that's hard to believe, but it seems to be true."
Hefner's relationship with his remaining two girlfriends - Marquardt, 35, and Wilkinson, 23 - may also be in flux. Hefner said Marquardt is in Europe filming the new Travel Channel series "Bridget's Beaches," and Wilkinson may soon move out of the mansion and get her own apartment. Hefner said E! is interested in spin-offs with all three women.
The Playboy mogul expects to maintain a business relationship and friendship with Madison, who - along with Marquardt - originally was one of seven girlfriends living with Hefner in 2001 after his separation from Kimberley Conrad. Wilkinson was later asked to move into the Playboy Mansion in 2004. Hefner said he may again seek out seven - or more - girlfriends.
"It's a big house," he said. "And I'm not going to live alone. I'm definitely not going to live alone."