The officer has been identified as Ian Hans Lichtermann, a 17-year veteran of the department, who the FOP says has never been in any trouble.
"This symbol, like it is a very clear, it's not like just a generic symbol of Germany, it's like specifically from the time around the Nazi-era," said Cornelius Moody, Philadelphia Coalition for Real Justice.
The group is demanding the officer be fired.
"The officer should be removed immediately. He shouldn't be paid while he is being investigated," said Moody.
The police administration says the matter is under review.
"Again, something we have to be mindful of, at no point in time does the department condone anything that may be interpreted as offensive, interpreted as hateful, or as being discriminatory in any form or fashion," said Philadelphia Police Lt. John Stanford.
On Thursday night, Mayor Jim Kenney said if the symbol turns out to be what it is alleged, that could be very offensive.
"If in fact that's what it is, it's a disturbing visual. That is a symbol of what people in WWII fought against, so we'll see how it goes and see where it turns out," said Kenney.
But Philadelphia FOP President John McNesby says the officer, who is of German heritage, was just trying to show his pride in the German-American Police Association, which he says uses a similar symbol, which he provided for us to see.
"He's a military veteran, he's a father, he's a great cop. On the other arm he has a tattoo of an American flag. Is that offensive? If he had a black panther on his chest or his back or his leg, would that be offensive?" said McNesby.
McNesby says Officer Lichtermann has an exemplary record with the department, with no disciplinary action ever taken against him.
"And now, he's being crucified, his family is being crucified. He had to change his phone number over a tattoo that he's had for a decade? Stop it. Come on," said McNesby.
As internal affairs conducts its investigation, McNesby says the FOP will be there to defend the officer from now until the very end - however long that may take.