The S&P 500's drop of 0.4 percent was its biggest daily percentage loss since May 20. The benchmark index fell for the second day in a row, after four straight record closing highs.
The selloff was broad. Every S&P 500 sector index except energy declined for the day.
Low volume and low volatility have marked recent sessions, leaving indexes to trade in a narrow range.
The World Bank's lower growth forecast provided investors with a reason to unload some stocks. Late Tuesday, the World Bank cut its global economic growth forecast for 2014 to 2.8 percent from 3.2 percent because of a harsh U.S. winter and the impact of the Ukraine crisis.
"It's pretty quiet. The only news investors are keying in on is the forecast of slightly lower global growth," said Dan Veru, chief investment officer of Palisade Capital Management LLC in Fort Lee, New Jersey, which oversees $4 billion.
"I think it's an excuse for some investors to take some money off the table."
The biggest drag on the S&P 500 was Bank of America Corp , down 2.1 percent at $15.59. The bank has reached an impasse in negotiating a multibillion-dollar settlement with the U.S. Department of Justice related to the bank's mortgage investments, according to The New York Times.
Investors turned cautious after the surprising primary election defeat of Eric Cantor, the No. 2 Republican in the House of Representatives, by an upstart candidate from the Tea Party movement.
The Dow Jones industrial average fell 102.04 points or 0.60 percent, to 16,843.88. The S&P 500 slid 6.90 points or 0.35 percent, to 1,943.89. The Nasdaq Composite dropped 6.07 points or 0.14 percent, to 4,331.93.
Even as the Dow and the S&P 500 retreated from recent gains, the PHLX semiconductor index kept up its rally. The SOX rose 0.5 percent, extending its winning streak to 15 days, its longest stretch of gains since the index was created about 20 years ago.
Leading the SOX higher, Micron Technology Inc rose 5 percent to $30.99 a day after Credit Suisse raised its price target on the memory chipmaker's stock to $50 from $30.
The CBOE Volatility Index rose 5.6 percent to 11.60 but remained well below its historical average of 20. In a sign of the market's low volatility, the 14-day Average True Range on the S&P 500 fell to 9.71, the lowest since February 2013.
Orexigen Therapeutics Inc shares sank 14.7 percent to $5.81 after the U.S. Food and Drug Administration delayed a decision on the marketing application for its obesity drug by three months.
Volume was once again below average. With just 5.2 billion shares changing hands on U.S. exchanges, Wednesday's volume was below the 5.76 billion average for the last month, according to data from BATS Global Markets.