There was no confirmation from Pyongyang that it had conducted a test, which it has been threatening for weeks.
Any test would be seen as another big step toward North Korea's goal of building a warhead that can be mounted on a missile. It would also be a bold signal from young leader Kim Jong Un, who took power in December 2011 following the death of his father, Kim Jong Il.
The South Korean Defense Ministry, which raised its military alert level after the quake, said it was trying to determine whether it was a test. Nuclear blasts can create tremors but they are distinct from those caused by natural earthquakes.
A nuclear test would also be a challenge to the U.N. Security Council, which recently punished Pyongyang for launching a long-range rocket seen as a covert test of ballistic missile technology. In condemning that December rocket launch and imposing more sanctions on Pyongyang, the council had demanded a stop to future launches and ordered North Korea to respect a ban on nuclear activity - or face "significant action" by the U.N.
A world nuclear test monitoring organization said it detected what it called an "unusual seismic event" in North Korea.
"The event shows clear explosion-like characteristics and its location is roughly congruent with the 2006 and 2009 DPRK nuclear tests," said Tibor Tóth, the executive secretary of the Comprehensive Nuclear Test Ban Treaty Organization. The DPRK refers to North Korea's formal name, the Democratic People's Republic of Korea.
"If confirmed as a nuclear test, this act would constitute a clear threat to international peace and security, and challenges efforts made to strengthen global nuclear disarmament and non-proliferation, in particular by ending nuclear testing," Tóth said.
Kim Min-seok, a South Korean Defense Ministry spokesman, told reporters that North Korea informed China and the United States of its plans to conduct a nuclear test. It was not clear when Pyongyang told Beijing and Washington.
The U.S. Geological Survey as well as earthquake monitoring stations in South Korea detected an earthquake near a site where North Korea conducted its second nuclear test in 2009, according to the government-funded Korea Institute of Geoscience and Mineral Resources.
"There is a high possibility that North Korea has conducted a nuclear test," said Chi Heoncheol, an earthquake specialist at the institute. Chi said a magnitude 3.9 magnitude earthquake and a magnitude 4.5 earthquake were detected in the North's 2006 and 2009 nuclear tests.
South Korean, U.S. and Japanese seismic monitoring agencies put the magnitude of Tuesday's quake between 4.9 and 5.2.
"We think it is possible it came about as a result of a nuclear test by North Korea from looking at past cases," Japanese government spokesman Yoshihide Suga said at a news conference Tuesday, noting that they were still gathering information.
"A natural earthquake normally starts with a smaller tremor followed by a larger one. This quake's strength was the same throughout," according to Yosuke Igarashi, an official at the Japan Meteorological Agency. He declined to elaborate on the length of the quake or other details, saying the agency was studying the data.
The United States and its allies have been on edge since North Korea said last month it will conduct its third nuclear test to protest toughened sanctions over the December rocket launch.
In Washington, the White House had no immediate comment on Tuesday's developments.
North Korea's politburo vowed to continue firing "powerful long-range rockets," but a statement by state media Tuesday made no mention of a nuclear test.
North Korea's National Defense Commission said Jan. 23 that the United States was its prime target for a nuclear test and long-range rocket launches. North Korea accuses Washington of leading the push to punish Pyongyang for its December rocket launch.
Last October, a spokesman from the commission told state media that the country had built a missile capable of striking the United States, but did not provide further details. A missile featured in an April 2012 military parade appeared to be an intercontinental ballistic missile, but its authenticity has not been verified by foreign experts.
Associated Press writers Youkyung Lee in Seoul and Yuri Kageyama in Tokyo contributed to this report.