The court heard arguments in a challenge to the program from a white Texan who contends she was discriminated against when the university did not offer her a spot in 2008.
The court's conservatives cast doubt on the program that uses race as one among many factors in admitting about a quarter of the university's incoming freshmen. The liberal justices appeared more supportive of the effort.
Justice Anthony Kennedy, whose vote could be decisive, looked skeptically on Texas' defense of the program. "What you're saying is what counts is race above all," Kennedy said. He has never voted in favor of an affirmative action program but has voiced support for diversity in education.
Twenty-two-year-old Abigail Fisher, the rejected student who sued, was among the hundreds of spectators at the arguments. Also in attendance was retired Justice Sandra Day O'Connor, who wrote the majority opinion in a 2003 case that upheld the use of race in college admissions.
Changes in the court's makeup since then, especially O'Connor's departure, could affect the outcome of the Texas case. Justice Samuel Alito, O'Connor's successor, has voted consistently against racial preferences since he joined the court in 2006 and appears likely to side with Fisher.
Among the liberal justices who looked more favorably on the Texas admissions system was Justice Sonia Sotomayor. She told Bert Rein, Fisher's Washington-based lawyer, that he was looking to "gut" the nine-year-old decision.
Justice Ruth Bader Ginsburg, who joined with O'Connor in the 2003 decision - Grutter v. Bollinger - said, "It seems to me that this program is certainly no more aggressive than the one in Grutter. It's more, in fact, more modest."
The university says the program is necessary to provide the kind of diverse educational experience the high court has previously endorsed. Along with race, the university considers community service, work experience, extracurricular activities, awards and other factors as it seeks to fill out its incoming classes. The bulk of its slots go to students who are admitted based on their high school class rank, without regard to race.
In 2008, the freshman class of more than 6,600 included 1,713 African-American and Hispanic students. Of those, 216 were admitted under the program that is being challenged.
Opponents of the program say the university is practicing illegal discrimination by considering race at all, especially since the school achieves significant diversity through its race-blind admissions.
No one on the court Wednesday expressed opposition to diversity in education. But Chief Justice John Roberts, Justice Antonin Scalia and Alito raised repeated objections to the affirmative action plan.
Roberts wanted to know how the university would determine when it had a "critical mass" of diversity on campus that would allow it to end the program.
Near the end of the session, he complained, "I'm hearing a lot about what it's not. I would like to know what it is."
Scalia, a dissenter in the 2003 case, mocked the university's efforts to have more minority students not just in the student body as a whole but at the classroom level.
"How do they figure out that particular classes don't have enough? What, somebody walks in the room and looks them over to see who looks Asian, who looks black, who looks Hispanic? Is that how it's done?" Scalia said.
After the argument concluded, Fisher read a brief statement outside in which she said she hoped the court would rule that race or ethnicity "should not be considered when applying to the University of Texas."
The university said in court papers that Fisher would not have been admitted even without a racial preference program.
The justices could sidestep the issue altogether. Texas argued that Fisher has no right to be in front of the court essentially because it can do nothing for her. She already has graduated from Louisiana State University, and the only money at issue in the case is her $100 application fee.
Justice Elena Kagan is not taking part, probably because she worked on the case at the Justice Department before joining the court.
A decision should come by late June.
Associated Press writer Jesse J. Holland contributed to this report.