In an exclusive statement to The Associated Press on Friday, Malcolm X's daughter, Ilyasah Shabazz, said Minaj's use of the picture "in no way is endorsed by our family."
"Ms. Minaj's artwork for her single does not depict the truth of Malcolm X's legacy," Shabazz's statement said.
Minaj apologized Thursday. She pulled the artwork after she was criticized on social media platforms for using the photo for the cover of her new male-bashing song, "Lookin (Expletive) (Expletive)." The title uses the N-word.
Minaj said the single cover was "never the official artwork."
Mark Roesler, CEO of CMG Worldwide - which is the business representative for the Malcolm X estate - called Minaj's use of the photo "dehumanizing."
"This is a family photo that was taken out of context in a totally inaccurate and tasteless way," said his statement on behalf of the family.
L. Londell McMillan, the attorney for the Malcolm X estate and the black leader's daughters - and estate administrators - Ilyasah and Malaak Shabazz, said in a statement Friday that those using the famous photo of Malcolm X need to remove it immediately.
"Failure to do so within the next 24 hours shall result in legal action," said McMillian, who is also the lawyer for Malcolm X's daughters, Attallah, Qubilah and Gamilah Shabazz. "The initial act and subsequent acts of distribution were improper and ill-advised. Any disparagement, infringement or disrespect of Malcolm X, and his name, image, likeness and proprietary rights will not be tolerated."
Minaj's representatives didn't return emails seeking comment.
Minaj posted the artwork Wednesday on her Instagram page and website. The photo shows Malcolm X holding a rifle as he was trying to protect his family from death threats; his home had been firebombed.
"I apologize to the Malcolm X estate if the meaning of the photo was misconstrued. ... I have nothing but respect (and) adoration for u," Minaj said on her Instagram page following the criticism.
The 31-year-old rapper-singer said later Thursday in an interview with radio station Hot 97 that she wrote the song to empower women because there are too many songs that attack females.
"It was almost parallel in my opinion because he has this big gun ready to shoot at a lookin' (expletive) bleep, and that's how I looked at it," she said. "I looked at it as this is one of the most memorable people in our history, in black history, who voiced his opinion no matter what, and I understand how my intent was overlooked and I definitely didn't want to offend his family or his legacy."
In the song's music video, Minaj is shooting guns.
Ilyasah Shabazz said Minaj's use of her father's photo is part of a larger problem in today's culture.
"Situations like the recent portrayal of our father on the album cover for Ms. Minaj's new single only highlight the fact that we as a society need to take more responsibility for what we're teaching our children," her statement said. "It is our family's hope that the true legacy and context of Malcolm X's life continues to be shared with people from all walks of life in a positive manner that helps promote the goals and ideals for which Malcolm X so passionately advocated."
Malcolm X was assassinated 49 years ago this month at age 39.