Bylsma won a franchise-record 252 games behind the bench but failed to produce a bookend to the championship he captured with stars Sidney Crosby and Evgeni Malkin in 2009. The Penguins were just 4-5 in playoff series since raising the Cup, with each loss coming to a lower-seeded team.
Pittsburgh's latest defeat came last month, when the Penguins fell to the New York Rangers in seven games in the Eastern Conference semifinals.
Rutherford, 65, replaces Ray Shero, who was fired three weeks ago. The move is a homecoming for Rutherford, who played goalie for the Penguins in the 1970s before spending 20 years with the franchise that began as the Hartford Whalers, moved to Raleigh, North Carolina, in 1997 and won the Stanley Cup in 2006.
Rutherford stepped aside in April when the Hurricanes promoted Ron Francis -- who helped Pittsburgh win consecutive Cups in 1991 and '92 -- to the GM's job. It's Rutherford's job to do the same with the Penguins.
"With some changes, they don't have to be sweeping changes, we can [win another Cup] in the near future," Rutherford said.
The Penguins also named Bill Guerin and Tom Fitzgerald as assistant GMs on Friday, and Jason Botterill was promoted to associate GM.
Although it's unlikely Rutherford will do much to mess with the core of Crosby, Malkin and Kris Letang, the Penguins will press on without Bylsma.
The firing marks the end of a difficult few months for the 43-year-old coach, who failed to lead Team USA to a medal at the Sochi Olympics in February.
The affable, open-minded Michigan native was a revelation when the Penguins promoted him from their American Hockey League affiliate in the spring of 2009, hoping his optimism would help a loaded team break out of a midseason funk.
It worked brilliantly. Pittsburgh roared to the Stanley Cup finals, edging Detroit in Game 7 to win its third title. Considering Crosby and Malkin were both in their early 20s at the time, more parades were expected.
Five years later, the wait continues. While Pittsburgh enjoyed nearly unparalleled success in the regular season -- including strolling to the Metropolitan Division title this season despite losing more than 500 man-games to injury -- the Penguins failed to reproduce the playoff magic of 2009.
Pittsburgh was 0-3 at home in Game 7s over the past five seasons, losing to Montreal in 2010, the Tampa Bay Lightning the following year and a 2-1 loss to the Rangers last month.
Bylsma had two years remaining on his contract, the product of an extension he received last June as a vote of confidence from Shero after an ugly four-game sweep at the hands of Boston in the Eastern Conference semifinals.
The extension came with a promise to adopt a more defensive-minded approach. The Penguins even brought in longtime NHL coach Jacques Martin as an assistant, an old-school yin to Bylsma's new-school yang.
The regular season went much like the four before it, with the Penguins streaking to the top of the standings behind a resurgent Crosby. The 26-year-old led the NHL in scoring with 104 points and missed only two games while becoming a near lock to win his second league MVP.
Crosby, however, struggled in the playoffs. He scored just once in 13 games, and while Bylsma consistently rose to the defense of the game's most popular player, there were signs of trouble. The two exchanged words briefly on the bench in the New York series, and Crosby's body language hardly suggested he remained close with the coach with whom he so happily raised the Cup a half-decade ago.
Rutherford said he is "very comfortable" with the control he has to name a replacement for Bylsma and hopes to have someone in place by the time free agency begins in July.
Coincidentally, the Hurricanes are looking for a new coach -- Francis' first big decision in his new job was firing Kirk Muller after three years -- and Carolina was mentioned as a possible landing spot for Bylsma under the guise that he would be ousted by Shero's successor.
The move by Rutherford -- who won 44 games in net for the Penguins from 1971-74 -- is the latest in a series of significant ties between the organizations.
Carolina has the longest active playoff drought among Eastern Conference teams. Its last postseason appearance came in 2009 -- when the Hurricanes were swept in the East final by a Penguins team that went on to win its only Stanley Cup under Shero's leadership, the crowning achievement of his eight years as Pittsburgh's GM.
Rutherford and Shero orchestrated the blockbuster trade of the 2012 NHL draft when forward Jordan Staal was sent to Carolina and reunited with big brother Eric in exchange for Sutter and prospects.
The Associated Press contributed to this report.