Riley, 61, comes from Oregon State, where he coached since 2003 after a three-year stint as coach of the San Diego Chargers.
"It is truly an honor to join the University of Nebraska family," Riley said in a statement released by Nebraska. "Though we love Corvallis and Oregon State, this is a once-in-a-lifetime opportunity to coach at one of the nation's most storied football programs and I can't wait to get started."
Riley will be introduced at a news conference Friday morning at Memorial Stadium. The coach will begin work immediately but will not be involved in bowl preparation.
Riley will receive automatic $100,000 increases in each of the five years of a contract that runs through Feb. 28, 2020. Riley had been the lowest-paid coach in the Pac-12, making a reported $1.5 million this year.
Former Nebraska coach Bo Pelini was paid $3.1 million this year.
Nebraska finished the regular season at 9-3 under Pelini, who was fired this week after seven seasons, and will learn its postseason destination Sunday.
Barney Cotton, an assistant under Pelini and a former Nebraska lineman, will coach the Huskers in their final game of the season.
Nebraska athletic director Shawn Eichorst chose Riley at the conclusion of a stealth search. Dozens of names surfaced in speculation, including former Ohio State coach Jim Tressel and Oregon offensive coordinator Scott Frost, but never the coach of the Beavers.
"There was one coach who fit all the characteristics that I was seeking to lead our tradition-rich football program," Eichorst said. "Mike Riley has a proven record of success, a sound approach to football and teaching, an understanding of the educational mission of our university and the integrity and values that we cherish at Nebraska.
"I have no doubt that Mike will assemble a tremendous staff and lead our student-athletes to win Big Ten titles and compete for national championships in the years ahead."
Riley was under contract with Oregon State, which finished 5-7 this season, through 2021. His contract stipulated that he would gain one additional year every time he went to a bowl game.
Riley employs a pro-style offense and is highly regarded for his development of quarterbacks. He has coached NFL QBs Matt Moore and Derek Anderson, and his starter this season, Sean Mannion, is the Pac-12's all-time leading passer.
"Although we are extremely disappointed of losing a person and coach the caliber of Mike Riley, we are happy for him regarding his new opportunity," Oregon State AD Bob De Carolis said in a statement. "We wish Mike, Dee and their family the best as his career begins a new chapter.
"Coach Riley has twice led a program, with integrity, to new heights and tremendous growth. The entire university owes him a well-deserved thank you."
The longest-tenured coach in the Pac-12, Riley took over an Oregon State program in 1997 that hadn't had a winning season since 1970. He left after two seasons to coach the Chargers, but not before his Beavers knocked off a nationally ranked Oregon in an overtime thriller in the 1998 Civil War game.
Riley returned to Corvallis in 2003 and had winning seasons in eight of the next 12 seasons. His Beavers famously knocked off No. 3 Southern California at home in 2006, No. 2 California on the road in 2007, No. 1 USC at home in 2008 and No. 9 Arizona on the road in 2010.
Riley is 93-80 as a college coach, a native of Wallace, Idaho, and a former defensive back under Paul "Bear" Bryant at Alabama.
The Associated Press contributed to this report.
Nebraska Hires Mike Riley
Joe Schad breaks down Nebraska's decision to hire Mike Riley as its next football coach.
Mike Riley A Terrific Hire For Nebraska
Brian Griese and Trevor Matich discuss Nebraska's decision to hire Mike Riley as its football coach and what he brings to the program.