The incident happened just before noon Wednesday outside the Frankford Avenue Health Center in the 4500 block of Frankford Avenue.
The girl was with her mother and her mother's friends at the time. Investigators say the man walked up to the girl and asked if she was 'old enough to date.'
"My mom was like 'No! And he went and he grabbed me," the girl told Action News. "He said it again, 'Are you old enough to date?' And my mom was like 'Get out of here, go, go, go!"
The man let the girl go then fled on foot toward Orthodox Street.
Witnesses say the man appeared to be mentally unstable or high on drugs.
Police are looking for surveillance video to see if the suspect was caught on camera.
"I don't want him to do that to another little person or girl," the victim said.
The suspect is described as a white man in his late 50s or early 60s. He was carrying a black school bag with one strap across his body. He was wearing a blue denim jacket and walked with a limp and was hunched over.
Anyone with information is urged to contact Philadelphia Police at 215-686-TIPS (8477), text a tip to PPD TIP (773847) or through PhillyPolice.com.