Victorino began walking toward the mound and Giants catcher Eli Whiteside stepped in front of him. Placido Polanco raced in from second base and was tackled by Whiteside, who along with Victorino was ejected. Ramirez also was tossed.
Giants first baseman Aubrey Huff was in the middle of the scrum and Phillies leadoff man Jimmy Rollins shoved San Francisco bench coach Ron Wotus as they exchanged words.
Victorino pushed plate umpire Mike Muchlinski trying to get back into the fray, then was held back by Wotus and Phillies hitting coach Greg Gross.
Philadelphia led 8-2. The favored Phillies were eliminated by the Giants in the NL championship series last fall as San Francisco went on to win the franchise's first World Series title since moving West in 1958.
The Giants took two of three in Philadelphia last week.