The collapse happened at 11:00 a.m. in the 1000 block of North 40th Street.
"There was a big noise, and a cloud of dust and stuff came up," said Darren Ross, resident of the home. "Shook me out of bed."
Rescuers say the outer wall of the second and third floors came down.
There were no injuries reported. The Red Cross is assisting the people who were displaced.
There was no immediate word on what caused the collapse.
Officials say the building must be demolished because of the collapse. Residents are going to be allowed back inside, to salvage whatever belongings they can.
Equipment will be brought in to the location Thursday night and demolition work will begin Friday.
