Castillo was fired Tuesday and replaced by secondary coach Todd Bowles.
Coach Andy Reid dismissed his longtime assistant who was surprisingly switched to defense after 13 seasons coaching the offensive linemen. Castillo was with the Eagles for 18 years, longer than any coach in franchise history.
"We're six games into the season and average isn't good enough," Reid said in a statement. "I know the potential of our team and insist on maximizing it."
Castillo has been under the microscope in Philadelphia, with nearly every move he's made being scrutinized intensely. He was promoted to defensive coordinator after a long search and with new defensive line coach Jim Washburn already in place running a wide-nine scheme.
"I put Juan in this position," Reid said at a press conference Tuesday at the team headquarters. "It didn't work out. I take full responsibility for putting him in that situation."
The Eagles (3-3) are on a bye this week. They wasted a 10-point lead with 5:18 remaining and lost 26-23 in overtime to Detroit after allowing Pittsburgh to rally for a winning field goal as time expired last week.
Castillo's defense held the Lions in check for three quarters, allowing just a pair of field goals. All-Pro wide receiver Calvin Johnson had one catch for 28 yards and Matthew Stafford was 7 of 21 passing.
But, according to defensive players, the Eagles inexplicably changed their game plan in the fourth quarter. They started blitzing more in an effort to pressure Stafford. Also, cornerback Nnamdi Asomugha didn't shadow Johnson the way he did the first three quarters.
Stafford picked apart the defense, Johnson had a big day and the Lions scored 20 points in the fourth quarter and overtime.
"I was on him most of the game," Asomugha said of covering Johnson. "I think when we got to the fourth quarter, there was a lot more trying to give him a different look."
Reid disputed that assessment a day later, saying "there wasn't a great change of scheme on what we did in the first three quarters."
Regardless, Castillo is gone.
Castillo joined the Eagles in 1995 as an offensive assistant under coach Ray Rhodes. He was promoted to tight ends coach in 1997, and then offensive line coach in 1998.
Bowles was 2-1 as the interim head coach for Miami last season. The former Temple star is in his 13th season as an assistant. He began his NFL coaching career with the New York Jets in 2000 as a secondary coach, spent four years in Cleveland, four in Dallas and three in Miami.