Almost unanimously, owners supported the commissioner Tuesday, as he handed down one of the harshest penalties in the history of U.S. sports.
"We stand together in condemning Mr. Sterling's views. They simply have no place in the NBA," Silver said at a news conference.
Sterling, 80, is banned for life from any association with the league or the Clippers, and fined $2.5 million - the maximum allowable under the NBA constitution. If three-fourths of the other 29 owners agree to Silver's recommendation, Sterling will be forced to sell the team he has owned since 1981.
A message left seeking comment at Sterling's business office hadn't been returned Tuesday afternoon. Team spokesman Seth Burton said in an email that the Clippers had no plans to issue a statement from Sterling on Tuesday.
Players and others cheered Silver's quick action, with union officials saying that if the league's punishment hadn't included a mandate for Sterling to sell the team, players were considering boycotting playoff games, including Tuesday's Golden State Warriors-Clippers matchup, the team's first home game since the scandal erupted.
"We wanted to be a part of this decision, and we wanted Adam Silver to know where we stood. And we were very clear that anything other than Sterling selling his team was not going to be enough for us," said Roger Mason Jr., the first vice president of the players' union.
Sterling's comments - which were recorded by his girlfriend and released by TMZ on Saturday - harmed the league, Silver said. Sponsors were threatening to abandon the NBA, and criticism was coming from fans on social media and even the White House.
Sterling criticized V. Stiviano - purportedly the female voice on the tapes - for posting pictures of her with black athletes Magic Johnson and Matt Kemp.
"It bothers me a lot that you want to broadcast that you're associating with black people. Do you have to?" Sterling asks the woman on the tape.
"Sentiments of this kind are contrary to the principles of inclusion and respect that form the foundation of our diverse, multicultural and multiethnic league," Silver said.
The NBA's longest-tenured owner keeps his team for now - and Silver said he didn't know if Sterling would fight to do so permanently.
But he can't attend games or practices, can't be involved in any personnel decisions or participate in board of governors meetings.
Just three days after the scandal broke, and hours before the Clippers hosted their biggest game of the season, Silver apologized to some of the league's black pioneers while meting out a punishment he believed would satisfy outraged players and fans.
Silver said the ban applied only to Sterling and there had been no discussions about whether he could sell to a family member.
Many owners supported Silver, and none of them publicly defended Sterling.
"We applaud the firm punishment handed out today by NBA Commissioner Adam Silver and appreciate the swiftness with which the NBA conducted its investigation," Warriors co-owner Joe Lacob said in a statement.
The fine will be donated to organizations dedicated to anti-discrimination and tolerance efforts, Silver said.
Sterling, with an estimated net worth of about $2 billion, did not comment, though Silver said he did not apologize for his remarks. Silver said Sterling confirmed that he was the person on the audiotapes.
Silver hasn't even been on the job three months and already had to face a crisis that threatened the league not only financially - with several companies ending or suspending their sponsorships of the Clippers - but more importantly, socially.
The NBA survived the Indiana Pacers' brawl with Detroit Pistons fans, and referee Tim Donaghy betting on games he officiated. But this brought a different level of outrage, particularly because the league could have done something sooner about Sterling, who has faced federal charges of civil rights violations and racial discrimination in his business dealings.
"This has all happened in three days, and so I am hopeful there will be no long-term damage to the league and to the Clippers organization," Silver said. "But as I said earlier, I'm outraged so I certainly understand other people's outrage."
After the announcement, the Clippers' website had a simple message: "We are one."
"We wholeheartedly support and embrace the decision by the NBA and Commissioner Adam Silver today. Now the healing process begins," the Clippers added in a statement.
Sterling's Clippers have been one of the most incompetent franchises in pro sports, and would have been home by now in most of their seasons. But this team is a title contender led by Doc Rivers, a black coach whom Sterling brought from Boston and paid $7 million a year.
Rivers canceled practice Monday and declined a meeting request from Sterling. He wouldn't address whether he would return next season if Sterling were still in control.
That might not be an issue if the owners vote to oust the owner.
Sterling is estranged from his wife and had been dating Stiviano, 31. In court documents, Stiviano describes him as a man "with a big toothy grin brandishing his sexual prowess in the faces of the Paparazzi and caring less what anyone else thought, the least of which, his own wife."
Silver said when he first heard the audio, he hoped it had been altered or was fake, but thought it was Sterling. And it doesn't matter if Sterling didn't realize he was being recorded, Silver said.
"Whether or not these remarks were initially shared in private, they're now public, and they represent his views."