Onil and Pedro Castro told CNN that they want Amanda Berry, Gina DeJesus and Michelle Knight to know how sorry they are for their ordeal.
"I'm just grateful they are home and out of that horrible house, and I'd just tell them I'm sorry for what Ariel done," said Pedro Castro, 50.
The brothers were initially taken into custody but released after investigators said there was no evidence against them. Brother Ariel Castro has been charged with rape and kidnapping and is being held on $8 million bond.
Pedro Castro says he was shocked to learn DeJesus was a victim, because they'd known her father for a long time, and Ariel even went to a vigil for her when she went missing.
"You go around like it was nothing? You even went to the vigil. You had posters. You give his momma a hug and you got his daughter captive?" Pedro recalled saying to Ariel.
Pedro Castro said he had been in the house but never noticed anything amiss. He said part of house was blocked off by curtains, and a radio or TV was always on.
Onil Castro, 50, said he wanted Ariel to "suffer in that jail to the last extent."
"I don't care if they even feed him," Onil Castro said. "The monster's a goner."
The brothers said they are worried that people will continue to associate them then with their brother's alleged crimes, even though police say they didn't know anything about it. They have received death threats since being released from custody.
"This has torn my heart apart," Onil Castro said. "This has killed me. I'm a walking corpse right now."
"I hope the world listens to us," Pedro Castro said. "You already got your monster, please give us our freedom."