Harbaugh, who was speaking at the team's annual banquet, said he's not even considering leaving Michigan.
"A lot of this talk is coming from our enemies," Harbaugh said. "They like to tell that to the recruits and to their families and try to manipulate them into going to some other school besides Michigan, but we know them as jive turkeys."
Harbaugh added the enemies to which he referred started with three coaches and "you know who I'm talking about."
"Say it like it is," he said. "That's the way it is."
Speculation about Harbaugh returning to the NFL started after the Los Angeles Rams fired Jeff Fisher on Monday.
"The team got all riled up," senior defensive lineman Ryan Glasgow said. "We don't want any enemies infiltrating the fortress. We knew. Based on how much Coach Harbaugh loves the program and loves Michigan, you know he's not leaving anytime soon, if ever."
Harbaugh left the San Francisco 49ers in December 2014 after four seasons to return to his alma mater in Ann Arbor. Reports about Harbaugh's interest in NFL jobs continued until he was officially introduced as Michigan's new head coach, and his name has been floated in conversations about NFL openings since then.
Harbaugh has not yet publicly addressed any of the rumors, but according to Glasgow, the coach held a "short and sweet" meeting with his players on Monday afternoon to tell them he didn't have any plans to leave the school.
Michigan players said they didn't need the reassurance, but they were happy to get it anyway.
"It's ridiculous," senior captain Jake Butt said Tuesday night. "If I'm not wrong, didn't he take less money to come here? I don't know if it's a recruiting pitch from other schools or whatever is going on. I can't even take it seriously."
Harbaugh makes an annual salary of $5 million at Michigan. He received an additional $4 million this season in the form of a no-interest loan for a life insurance policy.
Harbaugh is 20-5 in his two season at Michigan and will coach his sixth-ranked team in the Capital One Orange Bowl against No. 11Florida Stateon Dec. 30. He had a 44-19-1 record and won an NFC Championship during his four season with the 49ers.
Harbaugh adamant he's not leaving Michigan
ESPN Big Ten reporter Dan Murphy shares what Jim Harbaugh's reaction has been to rumors he will be the next Rams head coach.