Republicans scored big gains, taking Senate seats from Democrats in Pennsylvania, Wisconsin, Arkansas, North Dakota and Indiana. The net gain of 10 they needed for control of the chamber, however, eluded them.
With Republicans taking over the House, President Barack Obama will need a Democratic-run Senate to champion his legislative agenda.
Veteran Democratic Sens. Russ Feingold of Wisconsin and Blanche Lincoln of Arkansas lost their re-election bids.
But West Virginia Gov. Joe Manchin held off millionaire Republican John Raese to keep a Democrat in the seat held for half a century by the late Robert C. Byrd. And Sen. Barbara Boxer, D-Calif., won a fourth term despite a spirited challenge from Republican Carly Fiorina.
Those victories left Republicans no way to take the majority. They possibly could achieve a 50-50 split. But Vice President Joe Biden, the Senate's official president, would break ties in the Democrats' favor.
Tea party champions won high-profile races in Florida and Kentucky, spearheading a likely cadre of libertarian-leaning Republicans who will press party leaders to be more adamant about lower taxes, less spending and smaller government.
Rand Paul of Kentucky and Marco Rubio of Florida rocked the GOP establishment last spring by routing leadership favorites in party primaries. Then they beat back Democrats' efforts to paint them as too extreme, winning comfortably on Tuesday.
In Utah, tea party-backed Mike Lee also won easily after snatching the Republican nomination from Sen. Bob Bennett in March.
"Tonight there's a tea party tidal wave," said an exultant Paul.
Feingold, a three-term Democrat, lost to GOP newcomer Ron Johnson in Wisconsin. Best known for efforts to tighten campaign finance laws, Feingold was the only senator to vote against the so-called Patriot Act passed after the 2001 terrorist attacks, calling it a dangerous infringement on civil liberties.
Johnson, 55, made a fortune in manufacturing plastics. He wants to repeal the nation's new health care law, which he calls the greatest single assault on freedom in his lifetime.
Lincoln fell to GOP Rep. John Boozman in Arkansas, where Obama lost by 20 percentage points two years ago.
Conservatives said Lincoln, who won her first two Senate elections comfortably, was too close to Obama, while liberals said she wasn't loyal enough.
Republican Pat Toomey won a hard-fought race in Pennsylvania, beating Democrat Joe Sestak. The seat was held by Republican-turned-Democrat Arlen Specter, whom Sestak beat in the primary.
Indiana voters sent Republican Dan Coats back to the chamber after a 12-year absence. Coats, who spent a decade in the Senate before stepping down in 1998, defeated Democratic Rep. Brad Ellsworth. The seat is being vacated by Democrat Evan Bayh.
In North Dakota, Republican Gov. John Hoeven handily won the Senate seat that retiring Democrat Byron Dorgan held for 18 years. But Connecticut's attorney general, Richard Blumenthal, kept his state's open Senate seat in Democratic hands, fending off pro wrestling entrepreneur Linda McMahon.
Paul, who beat Democratic Kentucky Attorney General Jack Conway, is an ophthalmologist who had not sought office before. His father is Rep. Ron Paul of Texas, a hero to many libertarians. GOP Sen. Jim Bunning is retiring from the seat.
Rubio, a former Florida House speaker, is not a political newcomer. But he defied his party's establishment nonetheless, refusing to stand aside for Gov. Charlie Crist in the Senate race. Crist ran an independent effort, but Rubio comfortably defeated him and Democratic Rep. Kendrick Meek. The Florida seat's previous two occupants were Republicans who stepped down.
Christine O'Donnell, another tea party darling, lost to Democrat Chris Coons in Delaware. She also had won a stunning GOP primary victory, beating longtime Rep. Mike Castle, who was expected to top Coons. But she raised eyebrows with curious comments about witchcraft, the First Amendment and other topics, and failed to extend her popularity to the broader November electorate.
Tea partiers were hoping for up to three more Senate victories in western states. They included Nevada, where Sharron Angle hoped to beat Senate Majority Leader Harry Reid, and Colorado, where Ken Buck took on Democratic Sen. Michael Bennet.
But a tempestuous three-way race in Alaska threatened to let Democrat Scott McAdams win a once-hopeless race for GOP Sen. Lisa Murkowski's seat. Murkowski was running a rare write-in campaign after losing the Republican primary to another tea partier, Joe Miller.
In New Hampshire, Republican Kelly Ayotte kept her party in control of the seat being vacated by Judd Gregg. The former state attorney general defeated Democratic Rep. Paul Hodes.
Rob Portman won the Ohio Senate race, keeping a Republican in the seat that Sen. George Voinovich is vacating. Portman spent 12 years in the U.S. House starting in 1993. He later was budget director and then U.S. trade representative under President George W. Bush. Portman defeated Democratic Lt. Gov. Lee Fisher.
In Kansas, GOP Rep. Jerry Moran won the Senate seat vacated by Republican Sam Brownback, who was elected governor Tuesday.
And Rep. Roy Blunt kept Missouri's open Senate seat in Republican hands.
Easily winning re-election as expected were Sens. Mike Crapo, R-Idaho, David Vitter, R-La., Charles Grassley, R-Iowa, John McCain, R-Ariz., Jim DeMint, R-S.C., Richard Shelby, R-Ala., Tom Coburn, R-Okla., Richard Burr, R-N.C., John Thune, R-S.D., Johnny Isakson, R-Ga., Ron Wyden, D-Ore., Patrick Leahy, D-Vt., Kirsten Gillibrand, D-N.Y., Charles Schumer, D-N.Y., and Barbara Mikulski, D-Md.
Democrats technically hold 57 Senate seats, but two independent senators caucus with the party.