Brown died Monday night at the Wichita assisted living center where he lived, his law clerk, Nanette Turner Kalcik, said Tuesday.
During his long tenure, the senior judge in Wichita repeatedly tried to explain why he had not yet fully retired from the federal bench.
"As a federal judge, I was appointed for life or good behavior, whichever I lose first," Brown quipped in a 2011 interview with The Associated Press. How did he plan to leave the post? "Feet first," Brown said.
Brown was appointed as a federal district judge in 1962 by then-President John F. Kennedy.
In 1979, Brown officially took senior status, a type of semiretirement that allows federal judges to work with a full or reduced case level. But he continued to carry a full workload for decades later.
"I do it to be a public service," Brown said in the AP interview. "You got to have a reason to live. As long as you perform a public service, you have a reason to live."
His long tenure on the federal bench surpasses even that of Joseph Woodrough, a judge on the U.S. Court of Appeals for the Eight Circuit who, until Brown, had been the longest practicing judge in the federal judiciary when he died in 1977 shortly after turning 104.
In recent years, Brown's stooped frame nearly disappeared behind the federal bench during hearings. His gait was slower, but his mind remained sharp as he presided over a tightly run courtroom even after turning 104 last June.
It was not until March 2011 that Brown removed himself from the draw for assignment of new criminal case, and by the time he died he was no longer presiding over hearings.
Brown had asked his colleagues years before his death to notify him if at any point they felt he was no longer able to serve.
"I will quit this job when I think it is time," Brown said last year. "And I hope I do so and leave the country in better shape because I have been a part of it."
Brown - who was born on June 22, 1907, in Hutchinson, Kansas - was six years older than the next oldest sitting federal judge. At least eight other federal judges are in their 90s, according to a federal court database.
Brown started his career with the law firm of Williams, Martindell and Carey in Hutchinson. He graduated from the Kansas City School of Law, which later became the law school for the University of Missouri-Kansas City. Except for two brief breaks from the firm - one at age 27 when he was elected Reno County attorney and the other at age 37 when he joined the Navy - Brown spent his Hutchinson career practicing law there. In 1939, he became a partner.
He moved to Wichita at age 50 after receiving his first federal appointment as a bankruptcy judge in Wichita in 1958, and four years later was appointed a federal district judge in Wichita.
He outlived two wives and only moved into an assisted living center in recent years.