The court's 7-1 decision Monday leaves unsettled many of the basic questions about the continued use of race as a factor in college admissions.
Justice Anthony Kennedy, writing for the court, said a federal appeals court needs to subject the University of Texas admission plan to the highest level of judicial scrutiny.
The compromise ruling throws out the decision by the New Orleans-based 5th U.S. Circuit Court of Appeals, which upheld the Texas admission plan.
Kennedy said the appeals court did not test the Texas plan under the most exacting level of judicial review.
He said such a test is required by the Supreme Court's 2003 decision in Grutter v. Michigan upholding affirmative action in higher education.
"As the Court said in Grutter, it remains at all times the university's obligation to demonstrate, and the judiciary's obligation to determine, that admissions processes 'ensure that each applicant is evaluated as an individual and not in a way that makes an applicant's race or ethnicity the defining feature of his or her application,'" Kennedy said.
Justice Ruth Bader Ginsburg was the lone dissenter. "In my view, the courts below adhered to this court's pathmarking decisions and there is no need for a second look," Ginsburg said in a dissent she read aloud.
Justice Clarence Thomas, alone on the court, said he would have overturned the high court's 2003 ruling.
Justice Elena Kagan stayed out of the case, presumably because she had some contact with it at an earlier stage when she worked in the Justice Department.
Abigail Fisher, a white Texan, sued the university after she was denied a spot in 2008. She has since received her undergraduate degree from Louisiana State University.
The challenge to the Texas plan gained traction in part because the makeup of the court has changed since the last time the justices ruled on affirmative action in higher education in 2003. Then, Justice Sandra O'Connor wrote the majority opinion that held that colleges and universities can use race in their quest for diverse student bodies.
O'Connor retired in 2006, and her replacement, Justice Samuel Alito, has shown himself to be more skeptical of considerations of race in education.
Texas uses race as one among many factors in admitting about a quarter of the university's incoming freshmen. The school gives the bulk of the slots to Texans admitted based on their high school class rank, without regard to race. It automatically offers about three-quarters of its spots to graduates in the top 10 percent of their Texas high schools, under a 1990s state law signed by then-Gov. George W. Bush. Since then the admissions program has been changed so that now only the top 8 percent gain automatic admission.
Race is a factor in filling out the rest of the incoming class. More than 8 in 10 African-American and Latino students who enrolled at the flagship campus in Austin in 2011 were automatically admitted, according to university statistics.
In all, black and Hispanic students made up more than a quarter of the incoming freshmen class. White students constituted less than half the entering class when students with Asian backgrounds and other minorities were added in.
The university said the extra measure of diversity it gets from the slots outside automatic admission is crucial because too many of its classrooms have only token minority representation, at best. At the same time, Texas argued that race is one of many factors considered and that whether race played the key role in any applicant's case was impossible to tell.
The Obama administration, roughly half of the Fortune 100 companies and large numbers of public and private colleges that feared a broad ruling against affirmative action backed the Texas program. Among the benefits of affirmative action, the administration said, is that it creates a pipeline for a diverse officer corps that it called "essential to the military's operational readiness." In 2003, the court cited the importance of a similar message from military leaders.