Several justices, including some liberals who seemed open to gay marriage, raised doubts during a riveting 80-minute argument that the case was properly before them. And Justice Anthony Kennedy, the potentially decisive vote on a closely divided court, suggested that the court could dismiss the case with no ruling at all.
Such an outcome would almost certainly allow gay marriages to resume in California but would have no impact elsewhere.
Kennedy said he feared the court would go into "uncharted waters" if it embraced arguments advanced by gay marriage supporters. But lawyer Theodore Olson, representing two same-sex couples, said that the court similarly ventured into the unknown in 1967 when it struck down bans on interracial marriage in 16 states.
Kennedy challenged the accuracy of that comment by noting that other countries had had interracial marriages for hundreds of years.
There was no majority apparent for any particular outcome and many doubts expressed about the arguments advanced by lawyers for the opponents of gay marriage in California, by the supporters and by the Obama administration, which is in favor of same-sex marriage rights.
Kennedy made clear he did not like the rationale of the federal appeals court that struck down Proposition 8, the California ban, even though it cited earlier opinions in favor of gay rights that Kennedy wrote.
That appeals court ruling applied only to California, where same-sex couples briefly had the right to marry before voters adopted a constitutional amendment in November 2008 that defined marriage as the union of a man and a woman.
Several members of the court also were troubled by the Obama administration's main point that when states offer same-sex couples all the rights of marriage, as California and eight other states do, they also must allow marriage.
Justice Samuel Alito described gay marriage as newer than such rapidly changing technological advances as cellphones and the Internet, and appeared to advocate a more cautious approach to the issue.
"You want us to assess the effect of same-sex marriage," Alito said to Solicitor General Donald Verrilli. "It may turn out to be a good thing. It may turn out to be not a good thing."
Charles Cooper, representing the people who helped get Proposition 8 on the ballot, ran into similar resistance over his argument that the court should uphold the ban as a valid expression of the people's will and let the vigorous political debate over gay marriage continue.
Here, Kennedy suggested that Cooper's argument did not take account of the estimated 40,000 children who have same-sex parents. "The voices of these children are important, don't you think?" Kennedy said.