Officials have no immediate reports of injuries to humans or horses.
The National Weather Service said radar was tracking a confirmed tornado near the famed track and the University of Louisville campus about 8:10 p.m. Though no races are run on Wednesdays, there was simulcasting of races elsewhere, so people may have been there, said track President Kevin Flanery.
Churchill Downs spokesman Darren Rogers says at least five barns had roof damage and the chapel was hit. The barn damage was on the backside of the track where workers live in the dorms, Flanery said.
"It's a hell of a mess back here," said John Asher, spokesman for Churchill Downs.
The iconic twin spires above the clubhouse overlooking the finish line were not apparently damaged, Flanery said.
"Clearly we've got several barns with significant damage and we're just trying to make sure people and the animals are safe first," Flanery said.
Security guards were turning away reporters, citing danger from the loose horses. Vans were being brought in to move horses out of torrential downpours, and at least one barn was flooded by a water main break, Asher said.
The Kentucky Derby has been run for more than 130 years at the track. It is the first leg of the Triple Crown. Churchill Downs is in its spring meet, in which racing takes place Thursday through Sunday until July 4.
The track has a capacity to handle a crowd of some 160,000-plus for the Kentucky Derby.
No major damage has been reported on the university campus, which is sparsely populated at this time of year, said John Drees, a university spokesman. Drees said there were reports of power outages around campus. Dwight Mitchell, spokesman for Louisville police, said two buildings were damaged near the campus, though.
Eyewitnesses said they saw about a dozen power poles downed near the track and university. A weather service team will determine whether a tornado or straight line winds did the damage.
Weather service meteorologist Ryan Sharp said damage also was reported in the Shively area, west of downtown Louisville.
Storm sirens wailed in Kentucky's largest city as multiple tornado warnings were issued as the storm went through.
