Yahoo's overall revenue fell 6 percent in the last three months of the year to $1.266 billion, marking four consecutive quarters of eroding revenue. The company said that prices for both online display ads and search ads declined in the fourth quarter.
The company's shares were down 3.7 percent at $36.82 in after-hours trading on Tuesday.
"Normally you get better pricing in Q4," said BGC Partners analyst Colin Gillis. "The core business is shrinking."
Yahoo's efforts to revamp its slumping business have come to the forefront following Chief Executive Officer Marissa Mayer's decision to fire Chief Operating Officer Henrique de Castro this month, after only slightly more than a year on the job.
The move marked the first major change of plans since Mayer took the helm in July 2012, and underscored Yahoo's ongoing challenge to rekindle revenue growth.
Mayer has moved aggressively to kick-start the company with product makeovers, acquisitions and big media hires, including celebrity newswoman Katie Couric. But the ad sales business continues to struggle at a time when rivals such as Google , Facebook and Twitter are posting strong revenue growth.
Yahoo's stock has more than doubled since Mayer, a former Google executive, took the helm in July 2012. But analysts say much of the gain is due to aggressive stock buybacks and the expected IPO of Alibaba, in which Yahoo owns a 24 percent stake.
Yahoo repurchased $3.3 billion worth of its stock in 2013, the company said.
Yahoo's quarterly results also included some of Alibaba's financial results from the third quarter.
The Chinese company's revenue increased 51 percent year-over-year to $1.776 billion. While still robust, that growth rate was slower than the 61 percent clip that Alibaba delivered in the second quarter and the 71 percent growth rate in its first quarter.
"Alibaba was a big disappointment," said B Riley analyst Sameet Sinha, noting that the Chinese company's cooling revenue growth raised questions about whether it was losing its competitive edge, or whether some other factor might be the cause.
In a conference call with analysts on Tuesday, Mayer and Finance Chief Ken Goldman said that Yahoo's revenue would begin to grow again in 2014, thanks to the past year's investments and efforts boosting traffic to Yahoo's web properties.
But the company declined to provide a financial forecast for the full year, providing only an outlook for the current quarter.
Yahoo said that net revenue, which excludes fees paid to third-party websites, would range between $1.06 billion and $1.1 billion in the first quarter, a range whose midpoint matched the average analyst expectation of $1.08 billion.
In the fourth quarter, Yahoo's revenue from display ads was down 6 percent year-over-year, while the price per ad, excluding Korea, declined 7 percent.
Yahoo's fourth-quarter net income of $348.2 million rose from the $272.3 million earned in the year-ago period. Excluding certain items, Yahoo said it earned 46 cents a share, beating the average analyst expectation of 38 cents, according to Thomson Reuters I/B/E/S.