The yens advance towards a three-and-a-half-year high against the dollar also weighed on several exporters including Sony Corp But gains in several technology and telecom stocks helped prevent the market from falling further.
The benchmark Nikkei fell 0.45 per cent to close at 11,718.24, snapping a three-day, 480-point rally. The Nikkei earlier rose as high as 11,843.34, its best intraday level since June 4, 2002.
But the broader TOPIX index rose for a fifth straight day, adding 0.24 per cent to reach 1,179.17, its highest close since August 2001.
Overall market sentiment is still firm, but caution started emerging and short-term investors were seen starting to cash in profits, said Hiroyuki Nakai, Chief strategist at Tokai Tokyo Securities.
Also, there are growing concerns that the markets recent rally has boosted domestic-related stocks too much.
Steel makers met profit-taking, with JFE Holdings Inc, Japans second-largest steel maker, falling 3.45 per cent to 2,800 yen and Nippon Steel Corp losing 1.23 per cent to 241 yen. The steel sector subindex lost 1.52 per cent, after soaring more than 11 per cent last week.
General contractor Taisei Corp fell 5.05 per cent to 432 yen after hitting a 35-month intraday high on Friday.
Mid-sized general contractor Ando Corp fell 3.88 per cent to 248 yen, after having risen more than 40 per cent since the start of this month.
But NTT DoCoMo Inc, Japans dominant mobile phone operator, rose 2.7 per cent to 228,000 yen and Vodafone Holdings KK climbed 3.77 per cent to 248,000 yen.
Tokai Tokyos Nakai said more people were starting to see telecoms as a domestic business-oriented sector. He also said telecom stocks were seen as lagging steel and other domestic stocks that have marked hefty gains in recent sessions.
Trading edged down, with 1.467 billion shares changing hands on the first section, the lowest total since Monday last week and down from 1.69 billion shares on Friday. Gainers outnumbered decliners 822 to 631 on the first section.
Technology stocks ended mixed, despite some optimism before the market opened that stability on Wall Street might spur investors to switch their buying targets to heavily weighted techs from domestic business-oriented stocks.
Some investors also took to the sidelines ahead of key economic data, including industrial production, due out on Tuesday, analysts noted.
Japans industrial production likely fell 3.4 per cent month-on-month in February after strong export-led growth the previous month, a Reuters poll showed.
The data, along with employment figures, is due before the market opens.
Consumer electronics giant Sony fell 1.14 per cent to 4,320 yen and Hitachi Ltd shed 0.6 percent to 824 yen.
But office equipment giant Canon Inc rose 1.88 per cent to5,410 yen and NEC Corp picked up 0.93 per cent to 870 yen.
Hirokazu Toyoshima, a fund manager at Nikko Asset Management, said there were few attractive sectors in Mondays market as buyers had already hoarded domestic business-related stocks in the past few weeks and the markets rally in the past three days had also boosted technology stocks. Both domestics, like steel makers, and tech stocks are fairly valued now. Investors are now scratching their heads about which sector to invest in next, Toyoshima said.