"These are cities that have real challenges but also tremendous hope and opportunity," Duncan told reporters on a conference call Thursday.
The idea came from a meeting they had with President Barack Obama in May at the White House.
Education is high on Obama's priority list. He is seeking to boost achievement, keep kids from dropping out of high school and push every student to pursue some form of higher education.
The president has vowed to make the United States the world leader in the number of people who graduate from college.
He argues that students who do better in school will help themselves in a work force that increasingly depends on high-skilled jobs, and that the country will benefit as well.
Obama discussed education issues in an interview with Damon Weaver, an 11-year-old Florida student.
"On Sept. 8, when young people across the country will have just started or are about to go back to school, I'm going to be making a big speech to young people all across the country about the importance of education, the importance of staying in school, how we want to improve our education system and why it's so important for the country," Obama said.
Sharpton, the liberal Democrat and community activist, said teachers and administrators aren't the only ones responsible for improving schools.
"The parents need to be challenged with the message of `no excuses,"' Sharpton said.
Interviewed on NBC's "Today" show Friday, Gingrich and Sharpton were asked how they had agreed to work together on education in view of the many differences they've had on other issues.
"I think that he has it exactly right, that education has to be the No. 1 civil right of the 21st century and I've been passionate about reforming education," Gingrich said. "And we can't get it done as a partisan issue."
Sharpton said the time has come to "change the conversation ... to say we need to put everybody's hands on the table."
He said he believes that "if there's anything Americans should be mature enough about to have a decent conversation, it's the education of their children."
Gingrich applauded Obama for showing "real courage on the issue of charter schools." Obama wants to increase the number of charter schools, which have a controversial history and are a divisive issue for his party's base.
Charters get public tax dollars but operate free from local school board control and usually from union contracts, making them a target of criticism by many teachers' union members.
"I strongly believe that when you can find common ground, we should be able to put other differences aside to achieve a common goal," Gingrich said.