Williams had a back complaint she tried to conceal before Sunday, but Ivanovic noticed quickly that the best serve in women's tennis wasn't coming in as fast as expected, and started taking big swipes.
The No. 14-seeded Ivanovic ended Williams' bid for a sixth Australian and 18th Grand Slam title with a 4-6, 6-3, 6-3 victory that took the hottest of favorites out of contention at Melbourne Park and opened up one side of the women's draw.
"It's not easy playing such a champion ... but she is also just a human," Ivanovic said. "I just went out there swinging."
Williams hadn't lost a match since August, and had a phenomenal record of 78-4 in 2013. She came into the fourth round on the second-longest winning streak of her career and was playing her 70th match at Melbourne Park, a record in the Open era. Williams set the mark for most match wins (61) ever at the Australian Open with her third-round victory.
But she didn't move her feet well. And when she tried to step up the intensity in the third set - her grunts becoming louder and more frequent - Ivanovic matched her stroke for stroke.
"It wasn't the best," top-ranked Williams told a news conference, after hearing that her coach had leaked news of the back injury.
"I don't want to blame anything. I feel like Ana deserves all the credit," Williams added. "She played unbelievable today ... went for her shots. It's not like I gave her the match."
Ivanovic advanced to the quarterfinals here for the first time since her run to the 2008 final and will next play 19-year-old Eugenie Bouchard, who reached the last eight at a major for the first time after beating Australian wild-card entry Casey Dellacqua 6-7 (5), 6-2, 6-0.
The other quarterfinal in that half will feature two women who'll turn 32 next month, after two-time finalist Li Na beat No. 22 Ekaterina Makarova 6-2, 6-0 and No. 28 Flavia Pennetta upset No. 9 Angelique Kerber 6-1, 4-6, 7-5.
Novak Djokovic watched the second and third sets of fellow Serbian Ivanovic's win on TV, then continued his bid for a fourth straight Australian title with a 6-3, 6-0, 6-2 win over Fabio Fognini.
He next plays Stanislas Wawrinka, a 6-3, 7-6 (3), 7-6 (5) winner over Tommy Robredo. Djokovic and Warwinka played the longest Grand Slam match of last season in the fourth round here.
No. 3-seeded David Ferrer beat Florian Mayer 6-7 (5), 7-5, 6-2, 6-1 and next plays No. 7 Tomas Berdych, who beat No. 19 Kevin Anderson 6-2, 6-2, 6-3.
Ivanovic had never taken a set off Williams in four previous matches, and Williams had only ever lost once after winning the first set at the Australian Open.
The crowd was evenly divided at first but, sensing an upset, started backing Ivanovic, the former girlfriend of Australian golfer Adam Scott.
After dropping serve twice in the first set, Ivanovic didn't face another break point. She broke Williams three times, frequently standing well inside the baseline to receive.
Ivanovic had 33 winners, including 20 on her forehand. She also had some luck, when she drove a backhand off the top of the net and it looped onto the line.
"I had to remind myself all the time just to stay in the moment, because there were moments in the match where it could have gone either way," she said.
Ivanovic won her only major at the 2008 French Open, and briefly held the top ranking. But since then she'd only advanced past the fourth round once at 22 Grand Slams.
"I had to break a spell," Ivanovic said, "and what's the better place to do it than here against such a champion?"
Williams hit 22 winners but made 31 unforced errors, mostly on her backhand.
"I made a tremendous amount of errors, shots ... I haven't missed since the '80s," Williams said.
Desperate to halt momentum, Williams was given a warning for a time violation in her last service game, and fended off a match point when Ivanovic netted a return.
But Ivanovic served it on in the next game, finishing in less than two hours.
Williams didn't say exactly what the back injury was, but thought she'd get over it with some rest. She won her last Australian Open title in 2010, and hasn't been past the quarterfinals since then, her campaigns often derailed by injuries at the season's first major.
"I have done pretty well here, in general over the history," she said. "I feel like I'll win it again."