Derek and Maria Broaddus filed the suit against the town of Westfield after its planning board rejected their plan to raze the house and subdivide the land so they could build two houses. They say they can't live in the house because of the letters.
A lawyer for the town told NJ.com she cannot comment on the lawsuit.
The couple says that after buying the six-bedroom house for nearly $1.4 million in 2014, they received three threatening letters from a stalker calling himself "The Watcher."
Their attorney says they're caught in a situation they didn't ask for.