The suspects were reportedly from Tajikistan and crossed into the United States from the southern Mexico border in 2023.
ABC News has learned the suspects were initially allowed to enter the U.S. after being vetted and no national security issues were uncovered.
But sources say in recent weeks, authorities uncovered information indicating ties or affiliation with ISIS.
At least some were monitored and eventually all were detained.
Agents with ICE carried out the arrests over the last few days.
"Maybe they weren't using their real names. Maybe they were using different names," said Jack Tomarchio, the former director of intelligence for the Department of Homeland Security.
He says it's also possible federal authorities let them into the country to help lead law enforcement to an even bigger threat.
"We've had individuals that we knew were going to do something or planned to do something. We may have not immediately scooped them up because we wanted to see who they were affiliated with," said Tomarchio.
News of the arrests comes days after FBI Director Christopher Wray warned of heightened terror threats from both foreign and domestic actors. Wray has said the U.S. is facing accelerating threats from homegrown violent extremists as well as foreign terrorist organizations, particularly in the aftermath of the Oct. 7 attack by Hamas on Israel.
He said at one recent congressional hearing that officials were "concerned about the terrorism implications from potential targeting of vulnerabilities at the border." The Biden administration in August said that it had detected and stopped a network attempting to smuggle people from Uzbekistan into the U.S. and that at least one member of the network had links to a foreign terrorist group.
This also comes after Russia arrested migrants from Tajikistan earlier this year in connection with an attack on a concert hall near Moscow that killed more than 100 people.
Further details on the investigation in Philadelphia were not immediately available.
"The FBI and DHS will continue working around the clock with our partners to identify, investigate, and disrupt potential threats to national security," the agencies said.
Efforts are underway to deport the suspects. No terrorism charges have been filed and they are being held on immigration violations, said the people, who were not authorized to discuss the ongoing investigation by name and spoke to The Associated Press on condition of anonymity.
With the nation's 250th birthday set to be celebrated here, along with events like the World Cup in 2026, Tomarchio says he has no doubt these latest arrests have law enforcement thinking ahead.
"If some group of bad actors want to make a point or spoil our birthday party, we are going to be prepared for that. That's our job," he added.
The FBI and Department of Homeland Security released this statement on the arrests:
"Over the last few days, ICE agents arrested several non-citizens pursuant to immigration authorities. The actions were carried out in close coordination with the FBI's Joint Terrorism Task Forces. The individuals arrested are detained in ICE custody pending removal proceedings. As the FBI and DHS have recently described in public and partner bulletins, the U.S. has been in a heightened threat environment. The FBI and DHS will continue working around the clock with our partners to identify, investigate, and disrupt potential threats to national security."
The Associated Press contributed to this post.