Oliver died Tuesday after a sudden illness. She was 71.
"Sheila was a trailblazer in every sense of the word," Murphy said to reporters at the New Jersey State House.
Oliver, a native of Newark, was the first Black woman to hold a statewide elected office in New Jersey. She was also the first Black woman to lead the state assembly.
Oliver was serving as acting governor until she was hospitalized on Monday. Murphy and his family were on vacation in Italy at the time.
A cause of death for Oliver has not been released, and Murphy did not provide any details during Friday's press conference.
Gov. Murphy delivers remarks on passing of Lt. Gov. Sheila Oliver
"As governor, I relied on Sheila to shape our administration's policies on revitalizing our cities, expanding affordable housing, supporting our neighbors in need and so much more," Murphy explained.
The governor said Oliver will be remembered for many things, including her focus on housing assistance, her work on pandemic-era programs, and her impact on communities like Atlantic City and Trenton.
Murphy has directed all flags to fly at half-staff until September 4 in remembrance of Oliver.
Next week, a three-day celebration of life will be held for her.
Murphy said he is obligated to make a decision on his next lieutenant governor within 45 days of Oliver's passing. Right now, he said his focus is on honoring her life.
"The decision to ask Sheila to be my running mate was the best decision I've ever made," Murphy said. "God rest her soul and God rest her legacy."