The suit, filed Monday, claims New Line was required to pay 7.5 percent of gross receipts to Tolkien's estate and other plaintiffs, who contend they only received an upfront payment of $62,500 for the three movies before production began.
The writer's estate, a British charity dubbed The Tolkien Trust, and original "Lord of the Rings" publisher HarperCollins filed the lawsuit against New Line Cinema in Los Angeles Superior Court.
If successful, it could block the long-awaited prequel to the films.
Robert Pini, a spokesman for Time Warner Inc.'s New Line, declined to comment.
The films - 2001's "The Lord of the Rings: The Fellowship of the Ring," 2002's "The Lord of the Rings: The Two Towers," and 2003's "The Lord of the Rings: The Return of the King" - have reaped nearly $6 billion combined worldwide, according to the complaint.
The estimate includes everything from box office receipts to revenue from sales of DVDs and other products.
The plaintiffs seek more than $150 million in compensatory damages, unspecified punitive damages and a court order giving the Tolkien estate the right to terminate any rights New Line may have to make films based on other works by the author, including "The Hobbit."
Such an order would scuttle plans by New Line to make a two-film prequel based on "The Hobbit." "Rings" trilogy director Peter Jackson has already signed on to serve as executive producer on the project, which is tentatively slated to begin production next year, with releases planned for 2010 and 2011.
"The Tolkien trustees do not file lawsuits lightly, and have tried unsuccessfully to resolve their claims out of court," Steven Maier, an attorney for the Tolkien estate based in Britain, said in a statement. "New Line has not paid the plaintiffs even one penny of its contractual share of gross receipts despite the billions of dollars of gross revenue generated by these wildly successful motion pictures."
Maier also claims the film studio has blocked the Tolkien estate and the other plaintiffs from auditing the receipts of the last two films.
The lawsuit claims J.R.R. Tolkien established a trust through which he signed a film deal in 1969 with United Artists. After Tolkien's death, his heirs created the charity in the author's name.
Meanwhile, the original agreement terms were picked up by Hollywood producer Saul Zaentz, who produced an animated film in 1978 based on the "Rings" books, and eventually licensed the rights to make live-action films to New Line.
Lawyers for the plaintiffs said they have spent the years since the movies hit theaters trying to negotiate a settlement with New Line.
Other disputes over the film's earnings have surfaced in recent years.
In 2004, Zaentz sued New Line, claiming the studio cheated him out of $20 million in royalties from the film trilogy, which he optioned to New Line for a percentage of the movies' profits.
He and the film studio reached an out-of-court settlement a year later.
Jackson's production company also tangled with New Line in 2005 over profits from the films. A lawsuit was settled last year.
