Borel charged Castellano shortly after the $500,000 Breeders' Cup Marathon, angry over a dangerous mid-race move that endangered Borel and fellow rider Martin Garcia.
The two jockeys were near the weigh-in station next to the winner's circle when things grew heated.
The 43-year-old Borel pointed a finger at the 33-year-old Castellano. The 5-foot-1, 110-pound Castellano then took a swing at Borel, four inches taller than his rival. Security officials struggled to separate the two, holding an enraged Borel who urged them to let him go back at Castellano.
It took several minutes for Borel - his eyes bugging out and his face a deep shade of red - to be restrained.
In an interview with ESPN as he rode out for the Juvenile Fillies race, Borel said he and Castellano had patched things up. "I want to apologize to everybody," Borel said as he was interviewed aboard Tell A Kelly. "It's over. It's all good. We talked."
Borel was escorted to the paddock by eight security officers for that race. A fan shouted, "Kick some (butt), Calvin."
Borel declined to speak to reporters following the Juvenile Fillies, though he did stop to talk to trainer John Sadler after Tell A Kelly finished seventh. His agent, Jerry Hissam, said his client "just wants to ride."
Kentucky chief state steward John Veitch said officials will review videotape of the incident and interview both riders. If officials find either jockey at fault, penalties could range from a warning to a suspension. Veitch said neither rider had previously been disciplined for fighting.
The scuffle was a decided step out of character for the affable Borel, who has become a fan favorite at Churchill Downs for his rail-hugging rides and his ability to turn long shots into winners.
In May, he tucked Super Saver along the rail, his favorite spot, to win the Kentucky Derby, making him the first jockey to win three Derbys in four years.
And Borel nearly pulled off his own personal triple crown in 2009. Mine That Bird won the Derby, then Borel switched to filly Rachel Alexandra to win the Preakness before going back to Mine That Bird in the Belmont Stakes, finishing third.
Though jockeys have been known to tangle on the track during races, the postrace dust-up was jarring in a sport known for monied owners, big hats, seersucker suits and toothy grins in the winner's circle.
Not this time as a brawl more suited for a bar broke out in the track's most hallowed spot.
A few feet away from the chaos, Eldaafer owner Mike Iavarone held the championship trophy as security officials tried to restore order. Eventually, Borel's wife, Lisa, and brother Cecil each grabbed an arm and walked the three-time Kentucky Derby winning rider back to the jockeys' room, where he flung off his silks.
During the race, Castellano moved into the path of Romp and Martin Garcia, causing the horse to stumble. Borel and A.U. Miner were jostled as a result. Prince Will I Am later was disqualified in the race won by Eldaafer.
"I had pressure outside me," Castellano said. "I went for a hole and they said I took his lane. I don't know. I don't know."
Castellano returned to the track minutes later to ride in the Juvenile Fillies Turf while Borel remained in the jockeys' room to collect himself.
Prince Will I Am, who finished second behind winner Eldaafer, was disqualified and placed 10th while A.U. Miner was bumped up to third following an inquiry.
Borel was scheduled to ride for third time later Friday in the Ladies' Classic.
