Many investors said concerns about the "fiscal cliff" kept shoppers away from stores, suggesting markets may struggle to gain any ground until that issue is resolved. The CBOE Volatility Index or VIX, Wall Street's favorite barometer of investor anxiety, rose 4.46 percent, closing above 19 for the first time since Nov. 7.
A number of 2012's strongest performers advanced, a sign that portfolio managers may be engaging in "window dressing," a practice where market participants buy securities with big gains to improve the appearance of their holdings before presenting the results to clients. Bank of America Corp, which has more than doubled in 2012, added 2.6 percent to $11.54 on Wednesday.
Holiday-related sales rose 0.7 percent from Oct. 28 through Dec. 24, compared with a 2 percent increase last year, according to data from MasterCard Advisors SpendingPulse. The Morgan Stanley retail index skidded 1.8 percent while the SPDR S&P Retail Trust slipped 1.7 percent.
"With the 'fiscal cliff' hanging over our heads, it was hard to convince people to shop, and now it's hard to convince investors that there's any reason to buy going into year-end," said Rick Fier, director of trading at Conifer Securities in New York.
President Barack Obama is due back in Washington early Thursday for a final effort to negotiate a deal with Congress to bridge a series of tax increases and government spending cuts set to begin next week, the so-called "fiscal cliff" many economists worry could push the U.S. economy into recession if it takes effect.
Coach Inc fell 5.9 percent to $54.13 as the S&P 500's biggest decliner, followed by Amazon.com, down 3.9 percent at $248.63, and Abercrombie & Fitch, off 3.5 percent at $45.44. Ralph Lauren Corp, Limited Brands and Gap Inc also ranked among the S&P 500's biggest decliners.
The Dow Jones industrial average slipped 24.49 points, or 0.19 percent, to 13,114.59 at the close. The Standard & Poor's 500 Index shed 6.83 points, or 0.48 percent, to 1,419.83. The Nasdaq Composite Index dropped 22.44 points, or 0.74 percent, to 2,990.16.
J.C. Penney Co was a notable exception to the weakness in retail stocks, surging 4.4 percent to $20.75 as the S&P 500's biggest gainer. It was followed closely by Bank of America and Genworth Financial, which gained 3 percent on the day and 35.5 percent on the year.
"People want to show they own names like these, making them prime 'window dressing' candidates," said Wayne Kaufman, chief market analyst at John Thomas Financial in New York.
"Bank of America keeps going up even though it's overbought
and you'd expect a pullback at these levels. No one wanted it when it was under $10 a share, but they want it now." Volume was light, with only 3.96 billion shares having traded on the New York Stock Exchange, the Nasdaq and the NYSE MKT. Many senior traders were still on vacation during this holiday-shortened week and major European markets were closed for the day.
The S&P 500 has fallen 1.5 percent over the past three sessions, the worst three-day decline since mid-November. The Dow Jones Transportation Average, viewed as a proxy for business activity, fell 0.6 percent.
A Republican plan that failed to gain traction last week triggered the S&P 500's recent drop, highlighting the market's sensitivity to headlines centered on the budget talks.
During the last five trading days of the year and the first two of next year, it's possible for a "Santa rally" to occur.
Since 1928, the S&P 500 has averaged a gain of 1.8 percent during that period and risen 79 percent of the time, according to data from PrinceRidge.
"While it's unlikely there could be a budget deal at any time, no one wants to get in front of that trade," said Conifer's Fier, who helps oversee about $12 billion in assets.
"Investors can easily make up for any gains when there's more action in 2013."
Data showed U.S. single-family home prices rose in October, reinforcing the view that the domestic real estate market is improving, as the S&P/Case-Shiller composite index of 20 metropolitan areas gained 0.7 percent in October on a seasonally adjusted basis.
Decliners outnumbered advancers on the New York Stock Exchange by a ratio of about 2 to 1, while on the Nasdaq, more than five stocks fell for every three that rose.