The state removed Heath and Deborah Campbell's three small children from their home in January 2009.
A month earlier, the family drew attention when a supermarket refused to decorate a birthday cake for their son, Adolf Hitler Campbell. He and siblings JoyceLynn Aryan Nation Campbell and Honszlynn Hinler Jeannie Campbell have been in foster care.
A family court had earlier determined that there was insufficient evidence that the parents had abused or neglected the children. That decision was stayed until the appeals court could review it. On Thursday, the three-judge appeals panel determined there was enough evidence and that the children should not be returned.
The panel sent the case back to family court for further monitoring.
A gag order remains in place and the parties refused to discuss the decision.
Heath Campbell told The Associated Press last year that he believed the children were taken because officials felt they were in "imminent danger." He accused the state of removing the children because of their names and said government officials were relying on unproven accusations made by a neighbor and by an ex-wife who charged him with abusing her years ago.
The children's names and the birthday cake were not mentioned in Thursday's ruling. The court found that there were myriad other reasons that proved the need for continued protection services for the children.
According to court records, both parents are unemployed and both suffer from unspecified physical and psychological disabilities.
The court found that both parents were themselves victims of childhood abuse and said neither "have received adequate treatment for their serious psychological conditions."
Heath Campbell, 37, cannot read and Deborah Campbell dropped out of high school before finishing the 10th grade, according to court records.
In its ruling, the panel found the parents "recklessly created a risk of serious injury to their children by failing to protect the children from harm and failing to acknowledge and treat their disabilities."
The judges considered a typo-riddled note signed by Deborah Campbell and given to a neighbor. In it, Campbell says that if she were found dead, her husband was to blame.
"Hes thrend to have me killed or kill me himself hes alread tried it a few times. Im afread that he might hurt my children if they are keeped in his care. He teaches my son how to kill someone at the age of 3," the letter read in part.
Deborah Campbell later acknowledged writing the letter but claimed it was all a lie.
"She described her husband as 'a perfect guy"' according to court records.
The family made headlines when a ShopRite supermarket in Greenwich, near the family's home in Holland Township in west-central New Jersey, refused to decorate a birthday cake with their son's name.
A Wal-Mart in Pennsylvania wound up decorating the cake, but the resulting publicity put the family under media scrutiny. Heath Campbell said neighbors and others were harassing them, and local police reported a mailed death threat.