Klugman, who lost his voice to throat cancer in the 1980s and trained himself to speak again, died with his wife at his side.
"He had a great life and he enjoyed every moment of it and he would encourage others to do the same," son Adam Klugman said.
Adam Klugman said he was spending Christmas with his brother, David, and their families. Their father had been convalescing for some time but had apparently died suddenly and they were not sure of the exact cause.
"His sons loved him very much," David Klugman said. "We'll carry on in his spirit."
Never anyone's idea of a matinee idol, Klugman remained a popular star for decades simply by playing the type of man you could imagine running into at a bar or riding on a subway with - gruff, but down to earth, his tie stained and a little loose, a racing form under his arm, a cigar in hand during the days when smoking was permitted.
He brought a city actor ideal for "The Odd Couple," which ran from 1970 to 1975 and was based on Neil Simon's play about mismatched roommates, divorced New Yorkers who end up living together. The show teamed Klugman - the sloppy sports writer Oscar Madison - and Tony Randall - the fussy photographer Felix Unger - in the roles played by Walter Matthau and Art Carney on Broadway and Matthau and Jack Lemmon in the 1968 film. Klugman had already had a taste of the show when he replaced Matthau on Broadway and he learned to roll with the quick-thinking Randall, with whom he had worked in 1955 on the CBS series "Appointment with Adventure."
"There's nobody better to improvise with than Tony," Klugman said. "A script might say, 'Oscar teaches Felix football.' There would be four blank pages. He would provoke me into reacting to what he did. Mine was the easy part."
They were battlers on screen, and the best of friends in real life. When Randall died in 2004 at age 84, Klugman told CNN: "A world without Tony Randall is a world that I cannot recognize."
In "Quincy, M.E.," which ran from 1976 to 1983, Klugman played an idealistic, tough-minded medical examiner who tussled with his boss by uncovering evidence of murder in cases where others saw natural causes.
"We had some wonderful writers," he said in a 1987 Associated Press interview. "Quincy was a muckraker, like Upton Sinclair, who wrote about injustices. He was my ideal as a youngster, my author, my hero.
"Everybody said, 'Quincy'll never be a hit.' I said, 'You guys are wrong. He's two heroes in one, a cop and a doctor.' A coroner has power. He can tell the police commissioner to investigate a murder. I saw the opportunity to do what I'd gotten into the theater to do - give a message.
"They were going to do cops and robbers with 'Quincy.' I said, 'You promised me I could do causes.' They said, 'Nobody wants to see that.' I said, 'Look at the success of "60 Minutes." They want to see it if you present it as entertainment.'"
For his 1987 role as 81-year-old Nat in the Broadway production of "I'm Not Rappaport," Klugman wore leg weights to learn to shuffle like an elderly man. He said he would wear them for an hour before each performance, "to remember to keep that shuffle."
"The guy is so vital emotionally, but physically he can't be," Klugman said.
"We treat old people so badly. There is nothing easy about 80."
The son of Russian Jewish immigrants, he was born in Philadelphia and began his acting career in college drama (Carnegie Institute of Technology). After serving in the Army during World War II, he went on to summer stock and off-Broadway, rooming with fellow actor Charles Bronson as both looked for paying jobs.
He made his Broadway debut in 1952 in a revival of "Golden Boy." His film credits included Sidney Lumet's "12 Angry Men" and Blake Edwards' "Days of Wine and Roses" and an early television highlight was appearing with Humphrey Bogart and Henry Fonda in a production of "The Petrified Forest."
His performance in the classic 1959 musical "Gypsy" brought him a Tony nomination for best featured (supporting) actor in a musical.
He also appeared in several episodes of "The Twilight Zone," including a memorable 1963 one in which he played a negligent father whose son is seriously wounded in Vietnam. His other TV shows included "The Defenders" and the soap opera "The Greatest Gift."
In a 1987 interview in the New York Daily News, he said, "once I did three hourlong shows in 2½ weeks. Think we'd do that now? Huh! But then it was great. I did summer stock, played the classics. Me!"
Throat cancer took away his raspy voice for several years in the 1980s. When he was back on the stage for a 1993 revival of "Three Men on a Horse," The Associated Press review said, "His voice may be a little scratchy but his timing is as impeccable as ever."
"The only really stupid thing I ever did in my life was to start smoking," he said in 1996. Seeing people smoking in television and films, he added, "disgusts me, it makes me so angry - kids are watching."
In his later years, he guest-starred on TV series including "Third Watch" and "Crossing Jordan" and appeared in a 2010 theatrical film, "Camera Obscura."
Klugman's hobby was horse racing and he eventually took up raising them, too.
A horse Klugman co-owned, Jacklin Klugman, finished third in 1980's Kentucy Derby and fourth in that year's Preakness Stakes.
"I always loved to gamble," he said. "I never got close to a horse. Fate dealt me a terrible blow when it gave me a good horse the first time out. I thought how easy this is.
"Now I love being around them."
Klugman's wife, actress-comedian Brett Somers, played his ex-wife, Blanche, in the "Odd Couple" series. The couple, who married in 1953 and had two sons, Adam and David, had been estranged for years at the time of her death in 2007.
In February 2008, at age 85, Klugman married longtime girlfriend Peggy Crosby. His attorney Larry Larson wrote in an email that Klugman is also survived by two grandchildren and that memorial services have not been set.
In 1997, Klugman was sued by an ex-girlfriend, Barbara Neugass, who claimed he had promised to support her for the rest of her life. But a jury rejected her claim.
Biographical material in this story was written by former AP staffer Polly Anderson.