There is no guarantee the project that was originally expected to go online this year will continue.
IBM won't say how much money the project would have cost the company, but Monday's announcement followed several months of talks with university officials.
"As we moved forward in the project, increased cost from the final design and other changes made us come to the conclusion" not to continue, IBM spokeswoman Joanna Brewer said.
John Melchi, who is senior associate director at the university's National Center for Supercomputing Applications, said IBM's decision is disappointing but said that with the NSF waiting for a revised plan there's little time for the 120 or so people at the university who've worked on the project to do anything but keep working.
"When you look at the work you've put in the previous three or four years, it is disheartening," he said. "Honestly, there's a certain amount of energy here; people are inspired."
The NSF's National Science Board, a 20-member group of scientists, expects to have a revised plan to vote on as early as mid-September, NSF spokeswoman Lisa-Joy Zgorski said.
"Whether or not we move forward with a comparable alternative ultimately rests upon the decision of the National Science Board," she said.
When Blue Waters was announced in 2007 it was billed as a project to build what was expected to be, at least for a time, the world's fastest supercomputer. It was planned to run at sustained speeds of at least a thousand trillion operations a second, a measure known as a petaflop and a standard for speed that had long been sought.
The university bid on the project and was chosen by the NSF to build a computer that could study in new, much faster ways subjects such as the formation of galaxies and the effects of hurricane storm surges on land. IBM was selected as the vendor.
The fastest supercomputer at the time was another IBM product, Blue Gene/L, which had about a third of Blue Waters' expected capability.
Winning the bid was a prestige builder for both the university - which also used the new supercomputer as a recruiting tool - and for IBM. Melchi said Monday he doesn't believe the setback will cost the supercomputing center any staff.
NSF has agreed to spend $208 million on the project while the university and state of Illinois have pledged another $100 million. So far, about $160 million has been spent, the university and NSF said.
As part of the original contract, IBM is repaying the federal money it received, which the NSF says is about $30 million. The supercomputing center is returning IBM servers installed at the new National Petascale Computing Facility built on the campus for the project.
It's unusual for an NSF-funded computing project such as Blue Waters to lose a key partner midstream, "but not unprecedented," Zgorski said, adding that she wasn't sure if any projects were completed after running into similar problems.
Brewer did not disclose what went wrong, but said it didn't involve IBM's Power7 processors, the next generation of which is due to be released later this month.
Melchi said it's difficult to say how close the project is to completion - but it's "not close enough."
The trouble became obvious in April, he said, and IBM and the supercomputing center had tried since to find a solution before reaching what Melchi called an impasse.
As proposed, Blue Waters would not be the world's fastest because a computer in Japan known as the K Computer currently runs at a maximum speed of just over 8 petaflops - though much less if required to run for prolonged periods. Blue Waters' target is to run at a petaflop for sustained periods of time and run on much more complex problems than currently possible, Melchi said.
"The Blue Waters project is as relevant as it was in 2006," he said. "We're going to explore our options but, to be real open with you, we don't have a lot of time."