They said the actor was not in town and, therefore, would not attend the ceremony, which will see Sri Lankan President Mahinda Rajapaksa and Pakistan Prime Minister Nawaz Sharif, among other dignitaries, grace the occasion.
The sources did not divulge any further information.
The actor's move comes in the wake of a fringe student outfit staging a protest near his house last night, asking him to stay away from the ceremony in the light of Rajapaksa's participation.
All political parties in Tamil Nadu, including the ruling AIADMK and also the DMK, have opposed the invitation extended to Rajapaksa.
Earlier, during his election campaign in Chennai, Modi had taken time off and met Rajnikanth at his residence.
He had extended an invitation to Rajnikanth for the swearing-in ceremony today.