The fire started around 4 a.m. Saturday on the unit bock of West Washington Street.
Tisha Holloway said her stepmother lives in the home. She was asleep at the time, and so were her grandchild, son, and two unrelated children she was babysitting.
"My little brother didn't know that the other two kids were there. He just knew that his mom and his nephew were in there," said Holloway.
She said her brother was able to get his mom and the baby out, but the fire quickly engulfed the house, making it too dangerous to get to the other two children.
Mayor Gary Stevenson said police arrived to the scene initially and were told there were children in the burning home.
"The house is completely gutted. There are huge holes in the floor from what I can see being fire chief for many years in this town; it looks like it started in the living room," Stevenson.
Officers tried to gain entry into the house, but were driven back by the flames and smoke.
Firefighters responded within five minutes, the mayor said.
Crews put up ladders and were able to rescue two children from the second floor.
The children were initially taken to Crozer Hospital's burn unit and then transferred to St. Christopher's Hospital for Children.
Neighbors and family stood outside the home for hours Saturday in disbelief this could happen to the woman who lived there.
"She is the neighborhood mom, like she's here for everybody. She feeds everybody; summertime we have barbeques. She turns nobody away," said Holloway.
The Red Cross is helping the family with living arrangements. The community is also rallying to raise money for the beloved woman.
The fire marshal and prosecutor's office are investigating a cause for the fire.