He was accompanied by his Japanese counterpart Shinzo Abe when he went around the famous temple.
He was at the temple complex for about half an hour during which he enquired about the history of the eighth-century Buddhist pagoda.
The Chief Monk Mori took the Prime Minister around the complex having wooden temple including the five-storied main pagoda.
It stands 57 meters tall, making it the tallest pagoda in Japan, and has become a symbol of both the temple and Kyoto as it can be seen from many places across the city.
Monk Hasi, who was also accompanied Modi said: "We are happy that the Prime Minister visited us. It is a matter of pride for our temple. He has a big heart".
When Modi arrived at the temple, a number of Indians carrying the Tricolour were there to see him. He went and shook hands with the excited crowd.
Toji Temple is one of Kyoto's many UNESCO world heritage sites.