Modi, prime ministerial candidate for the Hindu nationalist Bharatiya Janata Party (BJP) and favourite to become India's next leader, is running on a platform of reviving an economy going through its worst slowdown since the 1980s.
But half way through a five-week campaign to win over the country's 815 million voters, some members of the BJP and its hardline affiliates are facing accusations of trying to whip up a partisan agenda.
Their statements have re-ignited concerns among religious minorities about a BJP government, which rivals say has a deep-seated bias against India's 150 million Muslims.
"This government belongs to those who have voted for it; this government belongs to those who have voted against it; this government belongs even to those who could not cast their ballot," Modi told the ABP News television channel.
"And the mantra of my government is absence of fear."
The comments came after Giriraj Singh, a leader of the Bihar state wing of the party, said those opposed to Modi would have to leave India and go to Pakistan after the BJP won the election and formed a government.
Modi said nobody could agree with Singh's comments.
In a Twitter post, he admonished his colleagues on the Hindu far right for railing against India's Muslims and liberals in the election campaign, dubbing their statements "irresponsible".
Television channels this week showed a video in which Praveen Togadia, a firebrand member of the Vishwa Hindu Parishad, a sister organisation of the BJP, was seen offering advice on how to prevent Muslims from buying property in Hindu-dominated areas.
Togadia denied that, saying he only asked Hindus to seek the help of police to resolve property disputes involving Muslims.
On Monday, a leader of the BJP's alliance partner in the Western state of Maharashtra said Modi would teach a lesson to Muslim rioters. Shiv Sena leader Ramdas Kadam made the comments at a joint election rally with Modi in Mumbai.
Modi himself is tainted by accusations that he turned a blind eye to, or even encouraged Hindu-Muslim riots in 2002 in Gujarat, the state he has governed for 13 years. More than 1,000 people, most of them Muslims, were killed in the violence.
He has always denied the accusations and a Supreme Court inquiry did not find evidence to prosecute him.
"These crocodile tears ... will not do," said Abhishek Manu Singhvi, a leader of the ruling Congress party, referring to Modi's Twitter posts criticising colleagues' statements.
"People know the truth."
While opinion polls predict Modi's BJP-led alliance will win the biggest chunk of the 543 parliamentary seats being contested in the election that ends on May 12, most of them show he will need new partners in order to secure a majority.
An anti-Muslim pitch would not only make it tougher for him to find partners, but could also drive away some middle-class voters whose support Modi is banking on to unseat Congress.
But some of Modi's colleagues remain defiant.
BJP leader Singh, who is contesting the election in the northern state of Bihar, said he stood by his statement. "I have said what I felt. I will give my explanation."
This month, the election commission banned one of Modi's top aides from election rallies on charges of making inflammatory speeches against Muslims.
The ban was lifted last week after the aide, Amit Shah, vowed not to use abusive or derogatory language. The commission said that it would monitor his campaigning.