BSkyB said he would be replaced as chairman by Nicholas Ferguson, the previous deputy chairman. Murdoch remains a member of the BSkyB board.
It was the latest setback for Murdoch, 39, a top executive in his father Rupert Murdoch's News Corp. global media empire. Murdoch - the former head of his father's British newspaper division - has been facing severe criticism as a result of the phone hacking scandal that brought down the company's News of the World tabloid in Britain and led to the arrests of scores of its journalists.
Murdoch said in a letter to the BSkyB board that he was "determined that the interests of BSkyB should not be undermined by matters outside the scope of this company."
"I am aware that my role as chairman could become a lightning rod for BSkyB and I believe that my resignation will help to ensure that there is no false conflation with events at a separate organization," he said.
Rupert Murdoch and News Corp. president Chase Carey said in a brief statement they were "grateful for James Murdoch's successful leadership of BSkyB."
"He has played a major role in propelling the company into the market-leading position it enjoys today and in the process has been instrumental in creating substantial value for News Corp. shareholders," they said. News Corp. owns 39 percent of BSkyB.
James Murdoch quit as chairman of News International, News Corp.'s British newspaper division in February. At the time, he indicated he was going to concentrate on managing parts of his father's extensive television businesses.
He has also stepped down from the boards of auctioneer Sotheby's and prominent pharmaceutical firm GlaxoSmithKline PLC but remains deputy chief operating officer of News Corp. and chairman and CEO, International, at the company.
News Corp. has been struggling for months to shake off the damage from the hacking furor, which derailed its plans to take charge of the remaining 61 percent of BSkyB last summer.
James Murdoch had been widely thought to be the heir to his father's media empire, and the proposed takeover of BSkyB would have boosted his profile as well as giving News Corp. full control of a lucrative broadcaster with an extensive sports and news franchise.
Rupert Murdoch shut down the News of the World tabloid in July amid a wave of bad publicity and plummeting ad revenue due to revelations its journalists had regularly listened to the phone voicemails of celebrities, politicians and crime victims in a quest for scoops. The scandal has spawned several major police investigations into the conduct of Murdoch's media empire, as well as a judge-led public inquiry into press ethics. Scores of Murdoch journalists have been arrested, and top Murdoch executives have resigned in the scandal.
Rupert Murdoch's companies have paid millions of pounds (dollars) to settle lawsuits from about 60 hacking victims, including actor Jude Law, soccer star Ashley Cole and singer Charlotte Church.
James Murdoch has repeatedly denied knowing about the widespread phone hacking at the News of the World, although his account has been contradicted by former associates.
In a letter to British lawmakers last month, James Murdoch acknowledged that he could have done more to investigate wrongdoing at the tabloid but insisted he had been misled and given "false assurances" by subordinates.
James Murdoch was re-elected chairman of BSkyB in November despite a groundswell of opposition. BSkyB said he won the support of more than 81 percent of shareholders who voted, while nearly 19 percent voted against him at the company's annual meeting.
BSkyB shares were down more than 1 percent at 674 pence Tuesday afternoon.
Richard Hunter, head of equities at Hargreaves Lansdown Stockbrokers, said BSkyB shares were falling because Murdoch's departure would "cause a bit more destabilization."
Hunter said Murdoch's resignation would not necessarily indicate a waning of News Corp. commitment to the broadcaster. Hunter sees it as a personal issue for James Murdoch: "I wouldn't read anything more into it strategically."