Carson, who was confirmed as HUD secretary last week and is the only black member of President Donald Trump's Cabinet, was talking about the work ethic and dreams of immigrants who came to the United States through Ellis Island in his first speech at the department.
"There were other immigrants who came here on the bottom of slave ships, worked even longer, even harder for less," Carson said as he walked across a stage holding a microphone.
A HUD spokesman declined to comment on Carson's statement. The department later tweeted: "This is the most cynical interpretation of the Secretary's remarks to an army of welcoming HUD employees. No one honestly believes he equates voluntary immigration with involuntary servitude!"
Carson has been considered a hero and motivational speaker in African-American communities for his accomplishments in medicine, and became a prominent speaker in conservative circles after entering politics.
The retired neurosurgeon has drawn criticism before when making comparisons to slavery. In 2013 he branded "Obamacare," the Affordable Care Act championed by President Barack Obama, "the worst thing that has happened in this nation since slavery."
Rana Hogarth, a history professor and expert on American slavery at the University of Illinois at Urbana-Champaign, said comparing slaves with immigrants was "inappropriate and wildly inaccurate." She said immigration "suggests a desire of a person to make the journey."
"I think that he's either misinformed or made a mistake," Hogarth said. "His beginning on Ellis Island tells me there's a major gap in his knowledge of how we talk about different ways people settled the United States and what circumstances they settled in United States."
Rebecca Scott, a University of Michigan law and history professor, said slavery in the United States was a "dramatically distinct form of migration," and that slavery made realizing the American dream much more difficult for captured Africans.
"That people had aspirations for their children regardless of how they were brought to the United States was certainly true," Scott said. "Their capacity to see their aspirations realized was starkly limited by slavery."
Carson spoke to HUD employees as he began his first full week leading the department. The Senate confirmed him last week.
The agency, with a budget of about $47 billion, provides housing assistance to low-income people through vouchers and public housing, enforces fair housing laws and provides development block grants to communities.
In his speech to a standing-room-only audience, Carson pledged to lead HUD with a "very big emphasis on fairness for everybody. Everything that we do, every policy. No favorites for anybody. No extras for anybody, but complete fairness for everybody."
He praised HUD employees for their dedication to HUD's "mission of really helping the downtrodden, helping the people in our society to be able to climb the ladder. Because to me, that really is what it's all about."
Democrats and housing advocates have expressed concern about his lack of government and housing policy experience. Carson said he plans to embark on a listening tour of communities and HUD field offices around the country to "see what works and what doesn't work."
Associated Press writer Jennifer C. Kerr contributed to this report.