The U.S. Geological Survey said the earthquake had a magnitude of 5.8 on the Richter Scale. The epicenter was half a mile deep and centered near Louisa, Va., about 40 miles from Richmond.
The quake, which hit at 1:51 p.m. and lasted only a few seconds, was felt up and down the Eastern Seaboard - from the Carolinas to Toronto.
The tips of three spires on the National Cathedral in Washington fell off, part of a building collapsed in Baltimore and a brick chimney crumbled atop a housing project in Red Hook.
That was the worst of it - except for the panic.
More than 12 million people may have felt the quake's sickening swaying, the USGS said.
"Wow, that was scary," said Nathan Buck, 41, a filmmaker who was editing a video on the 7th floor of a pre-war building in Harlem. "I saw my globe start to wobble. At first I thought - is it me? Then a pile of CDs came crashing down."
"I thought the building got bombed and it was 9/11 in the Bronx," he said. "I grabbed my son and held him as tight as I could. I have never been so freaked out in my life. I thought the building was coming down."
Office workers gather on the sidewalk in downtown Washington after the quake. (J. Scott Applewhite/AP)
Cellular service was disrupted as millions of people simultaneously reached for their phones to place a call or send a text message.
The city said calls to 911 spiked to 6,900 in the half hour between 1:30 p.m. and 2:00 p.m, most placed in the final 10 minutes. Normally, 911 receives 800 calls in a half hour.
No problems were reported with the MTA's bridges and tunnels or the subway, but flights at Newark and Kennedy Airports were briefly grounded as a precaution.
"There are no initial reports of major infrastructure damage, including at airports and nuclear facilities," said White House spokesman Josh Earnest after President Obamaconferred with his emergency management team
In Washington, D.C., most major buildings were evacuated. The quake shook the White House and was felt on Martha's Vineyard where President Obama was playing golf when it hit.
Two nuclear power plants near the epicenter in Virginia shut down automatically as part of a normal safety mechanism.