The singer surprised an assembly full of students at the Frank Sinatra School of the Arts in the borough of Queens on Thursday, appearing with Tony Bennett, who opened the school in 2001 through his Exploring the Arts program.
Joel performed songs on two different pianos onstage - he sang "New York State of Mind" on one and "She's Got a Way" on the other. In between his performances, he answered questions from students.
The first: "What do you think is one of your biggest mistakes?"
"My biggest mistake was signing a lot of contracts that I didn't know what they were about," said Joel, who released his first album in 1971. "I signed away a lot of my rights - record royalties, publishing rights, copyrights and it took me years to get that stuff back."
One male student asked for a hug as the audience cheered on, another had the 64-year-old sign his yearbook and a young girl got an autograph for her mother.
The crowd of 400 students brought a playful side out of Joel, who was lively with the students.
When asked who his favorite collaborator was, the Grammy winner answered: "Probably Elle Macpherson. That was a good collaboration," he said of the Australian model he dated in the 1980s.
One girl asked Joel if he would play "Uptown Girl."
"It's my favorite song," she gushed.
"It sounds like crap without harmonies and drums," Joel replied.
"I can be your harmony," she added to laughs.
Joel did not graduate with his high school class and instead was given a diploma 25 years later. He has made a number of visits to colleges in recent years - including a recent trip to Vanderbilt that went viral.
He said in an interview that his favorite moment in school was cutting class to go play the piano in the auditorium. He also said he was greatly affected by one of his teachers.
"I had a good chorus teacher and he encouraged me to become a musician. That's my greatest memory of school - an adult said, 'You should consider becoming a professional musician,'" he recalled. "I'd never heard (that) before in my life and that kind of changed my life."
Bennett's Exploring the Arts program supports 14 schools in New York and will launch three schools in Los Angeles this year. He and his wife, former teacher Susan Crow, kept Joel's appearance a secret since March.
"I had teachers texting me last night, 'Susan, who's coming?' My lips are sealed," Crow said.
Joel's interaction with the students was a memorable moment for Bennett, a Queens native.
"It's something that will stay with them forever and ever," he said.
Joel, who was born in the Bronx, said he was teased when he would get piano lessons since that teacher also taught ballet.
"I would walk by the guys on my block (and they'd say), 'Where's your tutu?' They'd knock the books out of my hand," Joel recalled, which earned a stream of "awws" from the students.
"But then I took up boxing," he said.