City prosecutors on Monday dropped charges against 52-year-old Linda Weston, her daughter and two others.
Federal prosecutors will now pursue charges that include hate crimes and two murder counts.
The defendants are due in federal court Monday afternoon for a brief appearance. A fifth defendant has been added to the case.
Authorities rescued five disabled adults from a locked Philadelphia basement in late 2011, and say they had been starved and beaten over the years. They also removed eight juveniles from Weston's care.
Weston previously served time for the slow starvation death of a man locked in her Philadelphia apartment.
