"He isn't the one that started it," Rivera said Monday. "If you're going to throw one guy out, you've got to throw the other guy out that started it. Especially because they [officials] saw the guy start it."
Williams and Saints linebacker Curtis Lofton both were cited for a personal foul in the melee that broke out after quarterback Cam Newton scored on a 2-yard dive to give the Panthers a 17-0 lead with 6:41 left in the first quarter.
The penalties were offsetting and only Williams was ejected.
"Unfortunately, we got a guy thrown out for being a little overaggressive," Rivera said. "That's unfortunate. But we stood up for ourselves and that's the approach. I appreciate that as far as our guys are concerned. We didn't walk away from anybody. We stood our ground."
The scuffle began when Newton got into a shouting match with defensive end Cameron Jordan. He eventually did in front of Jordan his "Superman" taking flight celebration (a thrusting gesture that looks similar to one that a pro wrestler makes as he's putting on his championship belt), followed by the "Superman" ripping open his shirt celebration.
At that point, Newton was pushed in the back by Lofton while continuing to jaw with Jordan.
Saints coach Sean Payton briefly addressed the fight Monday.
"Like any of those things, it starts off with just some shoving and then at some point it escalates," said Payton, who added that he didn't see the specifics on the coaches' tape. "But, look, the quickest way to stop that is not let them get in the end zone."
Rivera defended his quarterback, saying Newton wasn't taunting the Saints player, but understood no defender likes to see a celebration in his own end zone.
"I don't think the intent was to do it in the guy's face," said Rivera, whose team went on to a 41-10 victory to end a six-game losing streak. "The guy just happened to be there. You could say that about any time a guy does a celebration because there's 11 defenders on the field. A guy turns and does whatever they do, that's it.
"The biggest thing is Cam didn't use the ball, he didn't use a prop. He just did his thing. So be it."
Asked specifically if Newton's "Superman" taking flight routine could be interpreted as taunting, Rivera said: "No. Are you kidding? Some of the things that go on there and don't get called . . . "
Williams jokingly was asked if Newton had offered to pay any fine that may come from the NFL. He jokingly responded: "No, but he should, because he started it."
ESPN.com's Mike Triplett contributed to this report.
Brawl Erupts As Panthers Crush Saints
Cam Newton threw three touchdowns passes to lead the Panthers to a 41-10 win over the Saints.
Sunday Blitz: Panthers-Saints Recap
Jim Basquil and Ron Jaworski break down the Panthers' 41-10 win over the Saints.