The swap is subject to an upper limit of $15 billion and will be effective for three years, the RBI said.
The swap facility will enable the central banks to tap each other's foreign exchange and perhaps strengthen the currencies.
Both the yen and the rupee have fallen sharply against the dollar over the last few months.
The arrangement aims at addressing short-term liquidity difficulties and supplementing the existing international financial arrangements, as one of the efforts in strengthening mutual cooperation between Japan and India, the RBI said.
Up to 20% of the swap arrangement can be disbursed without an IMF-support programme.