Glassboro resident Tony Jones is one of those many people.
"Last year we purchased property in North Carolina because the taxes were much less. We're looking for a retirement home and we knew we didn't want to retire in New Jersey," said Jones.
Valorie Scott, a resident of Burlington City feels similar sentiments.
"As soon as the last of my children graduate, we're gone. It's because of the taxes and the economy here in New Jersey," said Scott.
Janice Sweeney of Mt. Laurel is with the 49 percent of residents that want to stay put.
"There's no place like home. I was born and raised here and I'm happy to be back. I'm close to Philadelphia, Washington, New York and all the culture. Everything's right here," said Sweeney.
Taxes were one issue people disliked most about the Garden State.
The poll did a quality of life survey looking at concerns like schools, the environment and safety.
In South Jersey, Cape May County rated the highest and Cumberland County the lowest.
"I think we just want New Jersey to be a state that we're proud of. One that everyone wants to stay in for the rest of their lives," said Burlington City Mayor James Fazone.
The poll found that 73 percent of New Jersey residents love their hometown, but not the state as a whole.
"I love the neighborhood where I live. I love the people I'm around but it may not be enough. I may not be able to afford it," said Patti Maltese of Jobstown.
Governor Chris Christie says he's trying to control taxes and resolve the issues that make so many residents want to leave the state.
"I can't worry about these numbers. What I have to worry about is changing the conditions here so that people will want to stay here," said Governor Christie.