Ling Ling was 22 years and seven months old, equivalent to about 70 human years, the Ueno Zoo said. It said he was the fifth-oldest known male panda in the world.
Ling Ling began losing his appetite and strength last August because of his old age, and recently suffered from heart and kidney problems, zoo official Motoyasu Ida said.
Keepers and visitors mourned the panda, which was the zoo's most popular attraction for more than 15 years. Public broadcaster NHK showed many visitors writing condolence messages, with some brushing away tears.
Ling Ling's portrait was displayed inside his cage, along with bouquets and offerings of his favorite bamboo shoots.
An autopsy found he died of heart failure, Ida said.
Zoo keepers said they found Ling Ling's body early Wednesday when they entered his cage to check up on him.
"He was resting at his favorite spot, a hole where he used to sit," Ida said.
Ling Ling died just one day after the zoo withdrew him from public view because of his worsening health.
Born at China's Beijing Zoo in 1985, Ling Ling came to Tokyo in 1992. He had traveled to Mexico three times in recent years for unsuccessful mating.
Ling Ling was the only giant panda owned by Japan, with eight other pandas elsewhere in the country all loaned by China, according to media reports.
With Ling Ling's loss, Ueno Zoo is without a panda for the first time since 1972, when the first panda couple arrived from China to mark the signing of bilateral peace treaty.
Ida said the zoo is currently consulting with the Foreign Ministry about obtaining another panda from China.
The regional newspaper Tokyo Shimbun reported Tuesday that Prime Minister Yasuo Fukuda may ask Chinese President Hu Jintao to consider the possibility of loaning a pair of pandas to Japan.
Giant pandas are one of the world's rarest animals, with about 1,600 living in the wild in China, mostly in Sichuan and Shaanxi provinces.
