Microsoft Corp. did not name a successor. The company said it is forming a search committee, which will include Gates, and Ballmer will stay on until a replacement is found.
"There is never a perfect time for this type of transition, but now is the right time," Ballmer said in a statement released by the Redmond, Wash., company.
Microsoft shares shot up 8 percent in morning trading following the news.
It's been less than two months since Microsoft announced a sweeping reorganization of its business in an attempt to reignite competition with faster-moving rivals such as Apple and Google.
Response to the newest version of Microsoft's flagship Windows operating system has been lukewarm. And Microsoft, along with other companies that thrived in the era of personal computers, are scrambling to transform their businesses as people come to rely more and more on smartphones and tablets.
In his statement, Ballmer noted that the company is moving in a new direction and needs a CEO that will be there for the longer term.
Microsoft, he added, "has all its best days ahead."
Ballmer, 57, met Microsoft founder Bill Gates in 1973 while they were living down a dormitory hall from each other at Harvard University. He joined Microsoft in 1980 to bring some business discipline and salesmanship to a company that had just landed a contract to supply an operating system for a personal computer that IBM would release in 1981.
Ballmer, a zealous executive prone to arm-waving and hollering, did the job so well that he would become Gates' sounding board and succeed him as CEO in 2000. He has worked at Microsoft for 33 years, matching the tenure of Gates, who left the company in 2008.
"It's a tad surprising, but every other business head has been rotated out," said BGC Financial analyst Colin Gillis. "They swapped out all their segment heads over the past few years. The only one they haven't changed is the CEO."
Though investors cheered the news on Friday, Gillis cautioned that it could be a "tough 12 months" for the company.
The obvious successor - former Windows head Steven Sinofsky - got booted by Ballmer, he said.
Sinofsky left the company shortly after the launch of Windows 8 last year.