"Shocking & unbecoming of a Prime Minister to mock secularism on foreign soil, when communal temperatures raised by his own party in India," Congress spokesperson Sanjay Jha said in a tweet.
Not mincing words, he said Modi was "beating the communal drum-beat in Japan" and the "mocking" of a serious issue of secularism is "ungracious, unacceptable, unexpected".
"The ferocious communal cat is out of the bag, officially! PM Modi ridicules secularists in India in Japan. We must be a laughing stock there," he tweeted.
His tweets drew reactions from twitterati in support of the ruling party.
Modi had an audience with Emperor Akihito yesterday at the Imperial Palace in Tokyo on the fourth day of his five-day visit to Japan to strengthen bilateral ties.
Addressing the members of the Indian community at a reception there, Modi said he had carried a copy of the Gita to gift to the Emperor.
"For gifting I brought a Gita. I do not know what will happen in India after this. There may be a TV debate on this.
"Our secular friends will create 'toofan'( storm) that what does Modi think of himself He has taken a Gita with him that means he has made this one also communal," the Prime Minister had said.
As the remarks were greeted with a round of applause, Modi hastened to add, "Anyway, they should also have their livelihood and if I am not there, then how will they earn their livelihood"