Officials say Thursday's early morning fire was the deadliest on record for the borough, which is about 30 miles southwest of New York City.
Relatives have identified the fifth victim as the children's grandmother, although authorities say they are still working to identify the dead and haven't released their names.
Authorities say four other members of the family were injured in the blaze. Only one remained hospitalized Thursday.
Fire Chief Thomas Scalera said Thursday the flames were so intense that they made rescuing people near impossible.
The Red Cross is providing the surviving members of the family with temporary lodging and emergency assistance.
