Jrue Holiday had 19 points and 12 assists and Thad Young scored 18 to help the Sixers win their third straight game.
DeMar DeRozan had 24 and Andrea Bargnani scored 22 for the Raptors, who are off to a 3-8 start. Jose Calderon, who had a season-high 18 assists in Toronto's win over Orlando on Sunday night, got 12 more to go with 13 points.
Down 87-80 early in the fourth, the Sixers came back to win their first game in which they trailed after three quarters. Nick Young's 3-pointer got Philadelphia to 92-89 with 5:11 left. After Richardson and Dorell Wright missed 3s that would've tied it, Young nailed another 3 to make it 92-all.
The teams traded baskets before Richardson scored on a driving layup, got fouled and made the free throw for a 99-96 lead with 1:54 remaining.
Richardson extended the lead to 102-96 with his fourth 3, and the Raptors didn't get any closer in the final minute.
For a while, it seemed the Raptors were going to run away with it.
After Holiday's dunk put the Sixers up 65-64 midway through the third quarter, Toronto scored 12 straight points. Bargnani started the run with a 16-footer and finished it off with a 3-pointer, giving the Raptors an 11-point lead that was the largest for either team.
But the Sixers answered in the fourth.
The Raptors built a 10-point lead early and led the entire first half until Philadelphia went on an 18-8 run late in the second quarter. Richardson hit a 3-pointer and scored on a fast-break layup off a steal to get the Sixers to 46-44.
Holiday hit a 3-pointer to give Philadelphia its first lead, 51-48, with 30 seconds left in the half. Calderon answered with a 3 to tie it going into the locker rooms.
NOTES: Richardson left the game briefly to be treated for a bruise after Kyle Lowry inadvertently stepped on his face in the third quarter. ... Lowry, who went to nearby Villanova, returned after missing six games with a foot injury. ... The Sixers completed a five-game homestand, going 3-2. ... The Raptors began a stretch of 10 of 12 games on the road. They are 1-5 away from home this season. ... Toronto visits Charlotte on Wednesday night. The Raptors already have three wins on the second night of back-to-back games. They were 4-16 in those situations last year. ... The Sixers were 0-4 when tied or trailing after three quarters.