The long-expected announcement left AT&T grumbling about a shortage of airwaves to meet growing demand, while scrappy competitor T-Mobile remains up for sale by German parent Deutsche Telekom.
The formal end of the deal was heralded by critics. No. 3 carrier Sprint Nextel Corp. had feared "an undeniable duopoly" between the proposed new entity and current leader Verizon Wireless. The two companies would have controlled over almost 80 percent of the cellphone market had the deal gone through.
"This result is a victory for the millions of Americans who use mobile wireless telecommunications services," Deputy U.S. Attorney General James Cole said. "A significant competitor remains in the marketplace and consumers will benefit from a quick resolution."
The Justice Department had sued on Aug. 31 to block the deal, and the Federal Communications Commission's chairman came out against it last month. That prompted the companies to withdraw their FCC application while they strategized their next move.
Sanford Bernstein analyst Craig Moffett said the announcement was "a bit of an anticlimax."
"This is like receiving the divorce papers for a couple that's been separated for years," he said.
AT&T's purchase of fourth-ranked T-Mobile, announced in March, would have made it the largest cellphone company in the U.S. AT&T is now the second-largest wireless carrier, with more than 100 million subscribers, behind Verizon Wireless, with 108 million. Sprint has 53 million, followed by T-Mobile at 34 million.
T-Mobile endured without much investment from its parent company and without the highest-end devices such as Apple Inc.'s iPhone. It offered value packages to customers who brought phones from other carriers. Regulators feared the loss of T-Mobile as a competitor would hurt consumers.
AT&T will now have to pay Deutsche Telekom $3 billion in cash as a breakup fee and give it about $1 billion worth of airwaves, known as spectrum, that AT&T doesn't need for the continued rollout of its high-speed "4G" network.
It will also enter into a roaming agreement with Deutsche Telekom so that AT&T's and T-Mobile's customers can use each other's networks.
AT&T will take an accounting charge of $4 billion in the current quarter.
In pulling out, AT&T said the government's attempts to block the deal do not change the challenges of the wireless phone industry. Cellphone companies have been clamoring for more airwaves to meet growing demand for faster downloads on smartphones and tablet computers.
The company said the deal would have solved that problem for a time, and without it, "customers will be harmed and needed investment will be stifled."
AT&T said it will continue to invest, and it called on the government to quickly approve its purchase of unused spectrum from Qualcomm Inc. and come up with legislation to meet the nation's long-term needs.
FCC Chairman Julius Genachowski said the commission agreed with AT&T that Congress should authorize an auction for additional spectrum. The FCC has proposed persuading television stations to give up their airwaves in exchange for sharing in auction proceeds.
Besides such an auction, AT&T could also buy spectrum from satellite TV operator Dish Network Corp., which could require an FCC waiver because it is licensed for satellite use.
Many people believed that AT&T had overstated the spectrum crisis.
AT&T already has an ample supply of unused wireless spectrum that it plans to use to expand its network over the next several years. And much of T-Mobile's spectrum is already in use, so the deal wouldn't have resulted in fresh airwaves becoming available. Furthermore, AT&T has made great strides in addressing network congestion in such cities as New York and San Francisco not by tapping its unused spectrum, but by upgrading its cell-tower equipment.
Moffett said AT&T's spectrum needs aren't so grave that it needs to make a large acquisition right away.
The decision to end the bid could be a bigger problem for T-Mobile than for AT&T. Deutsche Telekom has been eager to sell T-Mobile and isn't keen on investing more in the company, which has seen revenue decline slowly with the flight of higher-profit contract customers.
Besides missing out on many of the hottest smartphones, T-Mobile has stuck to updating its existing 3G network to achieve 4G speeds.
By contrast, competitors have moved to all-new networks that use LTE, or long-term evolution, technology specifically designed to carry data. Both AT&T and Verizon are building LTE networks, and Sprint intends to use the technology, too.
Still, Moffett believes it's too soon to write off T-Mobile, saying it has a good network with lots of room for more customers.
"I think they could surprise some people and be a more important force in the market than people are giving them credit for," he said.
AT&T's stock fell 8 cents to $28.66 in after-hours trading Monday after the news came out. Earlier, it closed the regular session down 11 cents.
Metz reported from San Francisco.