"My grandmother raised me and she said I always wanted a horse," he says.
Now, he's living the dream, with 14 horses at the Fletcher Street Urban Riding Club in Philadelphia's Strawberry Mansion neighborhood.
He founded the club in 2004 with a mission in mind.
"Give the kids something to do," he says. "We'll teach them how to ride."
He says most of the kids that come to the club's stables live in the neighborhood. Ferrell's son, Darrin Ferrell, helps out.
"I grew up around horses," says Darrin.
He says he started riding horses in Pennypack Park in Northeast Philadelphia after going with his dad to the stables.
"I was hooked, that was it," he says.
They teach kids how to ride after school and on weekends.
"The minute they get in here, they're comfortable," says Ellis.
For many kids, it's their first time on a horse and the club doesn't charge for lessons. Darrin says kids can pick up riding quickly, with one boy recently learning in about 15 minutes. Others, like Ellis's great-grandson, Aazim Ferrell, have been riding for years.
Darrin says it makes him feel proud to see Aazim participating in the club since "he's the next, next generation."
"Horses, they teach you a lot. It's like therapy," says Aazim Ferrell. "They'll trust you if you trust them."
The Ferrells say the kids that come to the club get attached to the horses and most have a favorite. While they're there, they also learn how to care for the horses, which includes feeding, cleaning stalls and grooming.
The club keeps the kids off the street and teaches them life skills through mentoring.
"I'm trying to teach them discipline," says Darrin.
Ellis says they want the kids to learn "a routine," so they do a lot of the work in caring for the horses.
"You can do horses and work and be productive in life," says Darrin Ferrell.
"If they learn to respect these horses, they'll learn to respect themselves," says Ellis.
Both father and son say the club's culture has transformed this section of Fletcher Street.
"They call it the safe haven," says Ellis.
"Once they open that gate, it's a different world," says Darrin.
Ellis says he just loves the club because the kids choose to come here.
"And it's always open to them," he says.
For more information on the Fletcher Street Urban Riding Club, visit: FSURC.com.