Mats Sundin scored both goals for Toronto, which lost despite a 56-32 shots advantage. All three goals on Andrew Raycroft were on rebounds.
Each team lost a player to injury in the first period. Toronto defenseman Carlo Colaiacovo reinjured his right knee, and Philadelphia forward Joffrey Lupul was too woozy to continue after colliding with big teammate Derian Hatcher.
Toronto fell to 1-4-1 since goalie Vesa Toskala went down with a groin injury.
Downie, who was suspended for 20 games after a preseason hit on Dean McAmmond in Ottawa, skated in his second NHL game since being recalled from the AHL on Wednesday and his fourth overall.
The 20-year-old native of nearby Newmarket outmaneuvered Colaiacovo for position and slammed a rebound past Raycroft at 10:04.
Carter converted a 2-on-1 break into a short-handed goal at 16:28.
Sundin slapped in a drive from the top of the circle to Niittymaki's right at 4:09 of the third period during a power play. The puck caught the far side of the net on the glove side. Sundin then scored his 20th off at 11:48 of the third.
Richards got the winner at 17:09 while Ian White sat in the penalty box for holding.
The 542nd and 543rd goals of Sundin's career moved him past Stan Mikita into sole possession of 25th place on the NHL's regular-season list. One more and he'll tie Maurice Richard.
It marked the 17th consecutive season Sundin scored at least 20 goals. Gordie Howe had 22 and Brendan Shanahan is working on his 19th. Marcel Dionne and Brett Hull did it 17 times.
A weak power play again hurt the Leafs. They had six advantages, compared to two for the Flyers, but they managed only Sundin's goal.
Notes: Colaiacovo played in his fifth game after taking months to recover from head and knee injuries. ... Raycroft hasn't won since Nov. 9. ... Leafs C Nik Antropov hasn't scored in 11 games. ... F Kyle Wellwood returned to Toronto's lineup after being a healthy scratch for two games. RW Wade Belak also was a healthy scratch as was D Andy Wozniewski, whose place was taken by D Anton Stralman. ... Philadelphia was without D Jason Smith (bruised calf), LW Scottie Upshall (ankle) and C Simon Gagne (concussion). Gagne hasn't played since Nov. 7. ... Toronto is 13-11-4 with Toskala in goal, and 3-7-4 without him.