The market also faced technical resistance as the S&P 500 earlier traded within a point of its record closing high set last month.
Minutes from the January meeting of the Federal Reserve's policy-setting committee showed that several policymakers wanted to hone in on the idea that their asset-purchase program would be trimmed in predictable, $10-billion steps unless there is a big economic surprise this year.
The statement doesn't deviate much from previous Fed communications, but market participants have been expecting the Fed to point to recent weakness in the economic data and reinforce their commitment to stimulating the economy.
"I think the rally that we've seen off the 1,750 low (on the S&P 500) was largely driven by the sense the Fed is going to slow down their retracement if it's necessary," said Uri Landesman, president of Platinum Partners in New York.
"Anything that puts a crimp in that belief is going to scare this market," he said. "This market is trading at nosebleed levels, and so it will not have a huge tolerance even for news that can be shaded in a bad direction."
Data on Wednesday showed U.S. housing starts recorded their biggest drop in almost three years in January. The seasonally-adjusted U.S. Producer Price Index for final demand rose 0.2 percent, giving no real indication of a broad pickup in inflation pressures.
The data was among a slew of recent economic reports affected by a severe U.S. winter, including a U.S. home builder confidence index on Tuesday, which suffered its largest ever one-month drop in February. The weather was also largely blamed for the sharp slowdown in hiring in December.
Some economists, however, lowered their first-quarter growth estimates on the back of the weak housing starts data. Goldman Sachs cut its first-quarter growth estimate by a tenth of a percentage point to a 1.8 percent annual rate. Barclays reduced its forecast by 0.3 percentage point to a 1.9 percent rate.
The Dow Jones industrial average fell 89.84 points or 0.56 percent, to end at 16,040.56. The S&P 500 slipped 12.01 points or 0.65 percent, to finish at 1,828.75. The Nasdaq Composite dropped 34.83 points or 0.82 percent, to close at 4,237.954.
The S&P 500 set an all-time closing high of 1,848.38 on Jan. 15, and came within a point of that level at its session high on Wednesday.
Tesla shares fell nearly 5 percent during regular trading hours, but jumped close to 12 percent after the bell. The electric car maker posted better-than-expected fourth-quarter results and said deliveries of its Model S electric sedan would surge more than 55 percent this year. The stock could hit a record high on Thursday.
Facebook shares were down over 2 percent in extended trading after closing at a record high, after news it will buy fast-growing mobile-messaging startup WhatsApp for $16 billion in cash and stock, plus an additional $3 billion worth of restricted stock units to WhatsApp's founders. Facebook, the world's largest social network, is looking for ways to boost its popularity, especially among a younger crowd.
Safeway Inc shares rose 3.2 percent in extended-hours trading after the second-largest U.S. mainstream grocery store operator said it is in talks about a possible sale of the company.
Chelsea Therapeutics International Ltd's shares soared 24.4 percent to $6.16 during regular trading a day after its drug Northera, which treats a rare form of low blood pressure associated with neurological disorders such as Parkinson's disease, won approval from the U.S. Food and Drug Administration.
Kay Jewelers parent Signet Jewelers said it would buy smaller rival Zale Corp for $21 per share in cash, valuing the mid-tier jeweler at about $690 million. The offer represents a premium of about 41 percent to Zale's close of $14.91 on Tuesday.
Shares of Signet Jewelers gained 18.1 percent to $93.65. Zale jumped 40.3 percent to $20.92.
About 6.96 billion shares traded on U.S. exchanges, roughly in line with the 7.06 billion average so far in February, according to data from BATS Global Markets.
Declining issues outnumbered advancing ones on the New York Stock Exchange by a ratio of 9 to 5. On the Nasdaq, about five stocks fell for every two that rose.