The banners, part of the "Heroes of Pottstown" project, feature photos of service members submitted by families.
But Robert Brown, whose father served in the Navy during World War II, said he noticed something missing.
"You're walking around, the flags look nice," he said. "There was a lot of people that weren't represented."
Brown said he began trying to add his father, Edward Brown Jr., to the display months ago.
"When we initially talked with them, it was way back in February," he said.
The project, launched by Elks Lodge 814, has been in place for about a year.
The lodge's president told Action News that all 74 banner spots, which are first-come-first-served, were filled almost immediately after the program was announced.
Families pay $200 and must provide a photo and proof of military service - requirements Brown said he has met.
Brown brought his concerns to the Pottstown chapter of the NAACP. Its president, Johnny Corson, said the absence of minority and female service members sends the wrong message.
"When you don't show minorities and people of color on the banners, it just makes it seem like the service that our Black, Spanish, Muslim, Jewish, female - because there's no females on these poles - their service meant nothing," he said.
To expand the display, the Elks Lodge would need the borough to make additional light posts available.
Brown and the NAACP say their issue is not with the lodge, but with borough leaders, whom they accuse of delaying the action of adding more light poles to the list where banners can be placed.
"You could go to Hanover Street. You can go to the park. There's so many places you can put banners up," Corson said.
The lodge president said she has arranged a meeting with the borough council president to discuss next steps. Brown said he hopes the outcome will allow his father and other underrepresented service members to be included.
"(America's) 250th is coming up. That's what makes it even more important about getting representation out there," he said.
The Elks Lodge said it hopes to expand the program. The borough manager and council president did not respond to requests for comment.