Among the horses killed were Kept Lady, a winner at Churchill Downs on Sunday, and another 3-year-old filly owned by Kenny Miller. He had celebrated his 33rd birthday along with the win.
"This business comes with a lot of highs and it comes with a lot of lows. And the low that you don't want is what happened today," he said. "You can survive losing races; you can go all year without winning races. But when you lose everything, it's a low that you don't want to go to."
The horses killed included one that survived the blaze but had to be euthanized. Four others in the barn at the time survived.
About 70 horses remain at Riverside, a former quarterhorse and harness track now used to stable and exercise horses, co-owner Mark Bowling said.
The fire, which started between 5 and 5:30 a.m. EST, was contained to one barn, which was destroyed. The barn was insured.
In January, six horses died in a fire at Riverside blamed on a vending machine electrical cord. Those horses were owned by John Hancock, a brother of Riverside manager Jack Hancock.
"It's just a terrible thing," Jack Hancock said Thursday. "The worst thing a horseman could ever hear is the word 'fire."'
Miller, of Evansville, Ind., was left contemplating whether he could start over in the business he loves. The recent win was Kept Lady's third, and Miller hoped the horse had plenty more left.
He spent every working day with Kept Lady and his other filly, Rodney's Lil Girl.
"They're family," he said. "You're attached to them. The only thing you look back and you say they aren't one of your kin folks. But it's close enough where it hurts. I've shed a lot of tears."
Trainer Steve Larue lost two of the five horses he had stabled in the barn. Three survived, including Countess Katie, a 2-year-old filly being treated for burns.
"I had tried to reach for her in her stall, but the smoke was so bad I couldn't see her," Larue said. "A couple minutes later she come walking out and her hair was on fire."
Larue said he expects Countess Katie to live, but doesn't know when or if the horse will race again. Countess Katie was scheduled to run in the first race at Churchill Downs on Friday.
Larue said many local trainers move their horses from nearby Ellis Park to Riverside Downs over the winter to train. He said the facility was well kept.
"We were in a nice barn, good people and everything," Larue said. "You couldn't ask for better people."
Associated Press sports writer Will Graves contributed to this report.