"Our focus is on the children, and under the current situation right now, we think the safest thing to do is take the children into state custody," he said.
Gurrolla, 30, was stabbed in her home Tuesday, just four days after giving birth to Yair Anthony Carillo, who was snatched by the attacker.
Nashville police said the baby was found in good health Friday night at a home in Ardmore, Ala., about 80 miles south of Nashville near the Tennessee line and a woman there was arrested in the case.
Earlier Saturday, officials said the newborn would stay with a foster family as authorities made arrangements for Gurrolla to be reunited with her son.
"This baby is a week old, and this child has spent half his life away from his family. I think it's time we reunite them," said My Harrison, a special agent with the FBI in Tennessee.
Johnson said officials made arrangements for Gurrolla to see her baby Saturday afternoon and hold him, and she brought her three other children - ages 3, 9 and 11 - with her. It's unclear when the children might return home.
Joel Siskovic, an FBI special agent in the Memphis division, said he could not say why the children were put into state custody. "As of now, there's no indication that there's an ongoing threat to the family," he said.
Tennessee Bureau of Investigation Director Mark Gwyn identified the arrested woman as Tammy Renee Silas, 39, of Ardmore. Federal authorities formally charged her Saturday with kidnapping. The Morgan County Sheriff's office said Silas was picked up by U.S. Marshals on Saturday morning, though it was not known where she was being taken.
The baby and Silas were found about 10 p.m. CDT in Ardmore, and Silas did not resist arrest, Gwyn said. Authorities said they had no word on a possible motive. Police in Nashville did not know if Silas has a lawyer.
The infant's mother mother told police a heavyset white woman with blonde hair arrived at her home posing as an immigration agent and attacked her with a knife.
Gurrolla told investigators that during the abduction, she heard the woman make a phone call and tell someone in Spanish words to the effect of "the job is done" and that the mother "was dying," said Siskovic, the FBI agent.
Siskovic said Silas took the victim's cell phone, which helped investigators locate Silas.
He would not comment further on the possibility that Silas was not working alone.
Silas' live-in boyfriend, Martin Rodriguez, said he was shocked by the arrest and didn't think she was capable of the crime. Speaking through an interpreter, he said, "The last thing that she said to me was, 'I am so sorry and I love you."'
Rodriguez said Silas told him she was adopting a baby from a cousin who had to go to jail, and was going to El Paso, Texas, to get the child. He said he picked Silas up from the Huntsville airport Tuesday and she had a newborn with her.
"She was acting normal around the baby and I didn't really see any difference, but I think she was happy," Rodriguez said from their one-story home, where a box of baby clothes for a boy overflowed in the dining room. "What woman isn't happy to carry a baby?"
Rodriguez said he met Silas, a contractor, when they both lived in Nashville. He said Silas is bilingual and was born in Tarrant County, Texas, where she had family.
At a Wednesday news conference, Gurrolla told reporters she had never seen her attacker before.
Speaking through an interpreter, Gurrolla said she did not see the woman take the baby because she ran to a neighbor's home for help. The neighbor, Eric Peterson, told The Associated Press that Gurrolla was "covered from her head to her toe with blood" with gashes on her neck and upper chest.
Gurrolla asked him to save her children from the "lady in the kitchen" who had a butcher knife. When Peterson got there, he saw a woman speeding away from the home. He brought Gurrolla's 3-year-old daughter back safely to his house, but found no baby, he said.
Officials believe Silas followed Gurrolla and her baby from a local office of the Women, Infants and Children program and to a Walmart store. "I think it's clear that she was targeting people at that location," Siskovic said.
Investigators got a break when they found that a video camera in the Walmart parking lot had captured the license plate of the car seen following the mother and baby, according to the arrest warrant.
Gurrolla's home was quiet Saturday morning, where a cleaning crew had been working inside. Some neighbors had placed flowers outside the house, and many neighbors said they were relieved Yair had been found.
Cathy Nahirny, a senior analyst for infant abduction cases at the National Center for Missing & Exploited Children, said there have been at least two other recent cases where an abductor used a ploy similar to the one used in this case.
"We need to get the word out to our immigrant communities," Nahirny said. "Anybody that claims they are from federal law enforcement agencies, you have the right and you should ask for photo identification."
Abductions of infants by strangers are rare, with only nine reported cases so far this year and five last year, according to the missing child center.
Nahirny said immigrant families have been targets of child abductions because of the assumption they will not tell police.
Gurrolla is Latina but her immigration status isn't clear. She was released from the hospital Thursday.
Associated Press writers Kristin M. Hall and Desiree Hunter contributed to this story.