"They turn you into a suspect," Weisman told the Seattle Times. "I just finally told them, 'I'm not going to go through it for three to five years. Forget it. ... That's the only reason, and it's a good enough reason."
Police Detective Mark Sommer confirmed Weisman's move to the Los Angeles Times and said Weisman has been difficult to track down.
"It is curious," Sommer said. "We'd like to talk to him about it."
The missing paintings were discovered by the family's longtime nanny at Weisman's Los Angeles home Sept. 3. Weisman was in Seattle at the time.
Police said there was no forced entry, a home alarm system was not on, and other valuable works of art were left untouched. There are no suspects in the case.
"Everything in the house was untouched, there wasn't even an ashtray overturned," Weisman told the Seattle Times.
Ten of the 40-inch-square portraits - believed to be worth at least $1 million apiece - feature famous athletes of the 1970s, including Muhammad Ali, Jack Nicklaus, Pele and Dorothy Hamill. The other is of Weisman himself, likely a commissioned portrait.
Weisman said he got to know Warhol when he was working as an investment banker in New York in the early 1970s.