Democratic Rep. Zoe Lofgren of California invited Colbert to appear before the House Judiciary Committee. But other Democrats weren't happy about her decision.
The committee chairman, Rep. John Conyers, D-Mich., asked Colbert to leave the room at the beginning of the hearing because the comedian had no expertise in farm labor issues or immigration policy.
House Majority Leader Steny Hoyer of Maryland told "Fox News Sunday" he thought the episode was more of an embarrassment to Colbert than to the House. But, he added, "I think it was inappropriate" that he testified.
Lofgren spokesman Pedro Ribeiro declined comment Sunday.