"If you look at it now, I think it gives the wrong message," said Kermit Roosevelt III, a professor at Penn Law.
The statue has stood at the entrance to the museum in New York since 1940.
READ MORE: Museum of Natural History to remove Theodore Roosevelt statue from front entrance
On Sunday, the city announced the statue would be coming down, calling it "problematic."
The museum said in a statement, "The statue itself communicates a racial hierarchy that the museum and members of the public have long found disturbing."
"I think the problem is you've got one white man on a horse and then a native person and an African person walking along, behind, below in this subordinate posture," said Roosevelt.
Roosevelt's namesake can be found all over Philadelphia in places like Roosevelt Junior High School in East Mount Airy and the Roosevelt Boulevard.
Kermit Roosevelt said there are so many memorials that represent the legacy well and he doesn't think that statue is one of them.
"That's a statue that people thought at the time was celebrating about him, but people's thinking at that time was very inflected by white supremacy," he said.
President Trump tweeted about the statue's removal saying, "Ridiculous, don't do it!"
Roosevelt said he hopes the former president is remembered for his work in conservation.
"He wanted a society where what's best about our natural parks, our natural resources was shared widely and available to everyone and I think he should be remembered as an egalitarian and a conservationist," he said.