Tempers flared after Ramirez hit Victorino in the back with a pitch. Victorino began walking toward the mound and Whiteside, the catcher, stepped in front of him. Placido Polanco raced in from second base and was tackled by Whiteside.
These teams have become quite the rivals of late after the Giants beat the favored Phillies in six games of the NL championship series last fall on the way to capturing the franchise's first World Series since moving West in 1958.
John Mayberry Jr. hit a two-run homer for his second clout in as many games and Vance Worley (8-1) won his sixth straight decision as the Phillies ran their winning streak to a season-best eight games.
Victorino and Hunter Pence also homered for the majors-best Phillies, who haven't lost since the Giants took two of three from them last week. San Francisco has dropped seven of eight during the same span.
Giants first baseman Aubrey Huff was in the middle of the scrum and Phillies leadoff man Jimmy Rollins shoved San Francisco bench coach Ron Wotus as they exchanged words. Victorino pushed plate umpire Mike Muchlinski trying to get back into the fray, then was held back by Wotus and Phillies hitting coach Greg Gross.
The umpiring crew met for several minutes once things settled down, then made the ejections.
Philadelphia led 8-2 at the time, then Pence homered for the second straight night in the seventh.
Whiteside hit a solo homer in the fifth for the Giants before being tossed the next inning.
Worley allowed two runs on seven hits, struck out six and walked one in seven innings, beating the Giants for the second time in three starts after tossing a three-hitter on July 26.
Giants starter Jonathan Sanchez (4-6) lasted 4 2-3 inning in his return from biceps tendinitis that landed him on the disabled list for 36 games. He is winless in his last five starts, going 0-3.
Sanchez took over in the rotation for struggling lefty Barry Zito, who went back on the disabled list with the same sprained right foot that shelved him earlier this year.
Giants manager Bruce Bochy went back to his regular lineup - with Carlos Beltran batting third and Pablo Sandoval cleanup - a day after Cliff Lee shut out the Giants 3-0. Sandoval hit a sacrifice fly in the first to put the Giants ahead, but the Phillies jumped on Sanchez in the fourth.
Victorino had a one-out homer and Mayberry connected four batters later. Raul Ibanez added an RBI groundout. Ramirez relieved after back-to-back singles by Polanco and Victorino in the fifth.
Orlando Cabrera was the lone Giant with two hits.
NOTES: Rollins stole his 25th base in the fifth and 26th in the sixth. ... Injured Giants C Buster Posey, the reigning NL Rookie of the Year who went down for the season in a home-plate collision with Florida's Scott Cousins on May 25, could be off crutches this weekend or early next week. He is on one crutch after tearing three ligaments in his left ankle and fracturing a bone in his lower leg. "He's doing well. He's real close to being able to put all his weight on his leg. He seems to be in a great frame of mind with the progress he's making." ... Zito will pitch in a rehab outing Sunday for Class-A San Jose. Bochy said he could go four or five innings depending how he feels. ... RHP Matt Cain goes Saturday for the Giants, trying to beat Philadelphia for the second time in as many starts. His win at Philly last week was his first ever vs. the Phillies. ... The Phillies will send LHP Cole Hanmels to the mound against the Giants on Saturday. He's 4-3 against the Giants in nine starts, including a loss in Philadelphia. He is 1-2 over his last four starts. Cody Ross has four home runs in 32 at-bats against Hamels, while Beltran has three in 36. Aaron Rowand is 4 for 19 with six strikeouts. Ryan Howard is 2 for 13 against Cain, both hits home runs. Cain and Hamels faced off in Game 3 of the NLCS, with Cain shutting out the Phillies over seven innings in a 3-0 Giants win.