Often described as the country's answer to a royal family, with the added tragic glamour of the Kennedys, the dynasty gave India its first prime minister, the empire-beating barrister Jawaharlal Nehru.
His daughter, Indira Gandhi, and grandson, Rajiv, both held the post subsequently, and both were assassinated.
Friday's electoral humiliation risks consigning the family that has ruled India for most of its 67 years since independence to political oblivion, as Modi, who cast the Gandhis as elitist throughout his campaign, looks to sideline them for good.
By some measures, the family was in decline long before the parliamentary election; it has not won a majority in decades.
But the sheer scale of Modi's victory this time around, dealing the Gandhis' Congress party its worst election defeat ever, underlines how deeply that decay had spread.
"The Congress has done pretty badly, there is a lot for us to think about. As vice president of the party I hold myself responsible," said Nehru's great-grandson Rahul Gandhi, who was consistently outshone by Modi on the campaign trail.
Gandhi was flanked by his mother, Rajiv's widow Sonia, who also delivered a brief concession speech, taking a swipe at Modi's Hindu nationalist policies that she says are divisive.
"We hope the government that will be formed in the centre will not compromise the unity of Indian society and the interests of the country," she said, while also shouldering blame for the debacle.
Shy scion Rahul's bid to keep Congress in power for a third consecutive term was called lacklustre even by allies, and his speeches at rallies up and down the country in recent months were a far cry from Nehru's legendary rhetoric.
Compare that with Modi's electrifying campaign, during which he repeatedly derided Rahul, 43, and Sonia for keeping India poor, and the house of Gandhi looked vulnerable.
Few would write off the clan completely. Sonia, the power behind the prime ministerial throne occupied by Manmohan Singh, delivered Congress its worst result to date in 1999. She then led the party to victory in the next two elections, and a year ago Forbes ranked her as the world's ninth-most-powerful woman.
The party faithful, while reeling as partial results on Friday showed Modi's Hindu nationalist Bharatiya Janata Party (BJP) led in more than six times as many seats as Congress, were quick to rally around the wounded Gandhis.
"Giving up on the Gandhis at this juncture would be the most stupid thing the Congress could do," Mani Shankar Aiyar, a former minister and family loyalist, told Reuters.
For those in the party looking for fresh blood, the search ends with Rahul's charismatic sister, Priyanka, who had an important backroom role in the campaign.
"Party men would embrace her with both arms the moment she wants to join," Aiyar said.
"HE WILL DEFANG THEM"
Leaders of both Congress and Modi's BJP said they believed Modi would seek to loosen the dynasty's grip on India.
They pointed to his home state of Gujarat, where he has systematically purged rivals from institutions and won three consecutive terms, capitalising on his pro-business policies.
"He will defang them politically. Look at what he did in Gujarat: he has just reduced the Congress to a non-player," said Kanchan Gupta, member of the BJP's national executive committee.
Modi has questioned Sonia's non-native roots - the widow of Rajiv was born in Italy. On Thursday, a BJP ally called Rahul a "foreigner", even though he was born in New Delhi.
During the election, Modi contrasted his humble past as the low-caste son of a tea seller with Rahul's privileged and cloistered life in plush districts of the capital.
In one recent newspaper interview, Modi said the family's leadership could come under threat if the party failed to win 100 of parliament's 543 seats. The party won just 42 seats, according to trends on Friday afternoon.
Congress sources said they were worried Modi planned to drive the family out of politics.
"We're assuming this will be one of his priorities," said a Congress strategist who is close to the Gandhis. "The family is not worried, but the party is," he said.
The scale of Modi's antipathy to the Gandhis was on display at the start of the campaign last year, when he launched the construction of the world's tallest statue, a $338 million, 182-metre tall homage to Sardar Vallabhbhai Patel, Nehru's deputy and interior minister, who was often at odds with him.
Modi, a Hindu nationalist, sees Patel as a symbol of an India imagined without the dynasty, who would have led the country down a different, right-wing path if he had not been thwarted by the socialist and atheist Nehru.
The Congress party has lost power several times since Nehru's era. After his daughter Indira Gandhi led the party to a crushing defeat in 1977, the prime minister who replaced her humiliated the family with arrests and investigations.
But that caused a backlash of sympathy among the public that helped propel her back to power with a landslide majority three years later.
Modi is not expected to follow the same course of using tax and police agencies to harass the Gandhis. In a campaign speech last month, Modi said he did not believe in "vendetta" politics or witch-hunts.
"The Janata Party government was in too much of a hurry," said Gupta. "Modi is too savvy to be seen to be openly persecuting the dynasty."
Going by his record in Gujarat, Modi prefers to move methodically against his opponents - often with the help of close aide Amit Shah, who held multiple posts in the Gujarat government and led Modi's campaign in Uttar Pradesh, politically the most important state, during the election.
After Modi took office in Gujarat in 2001, Shah, a former stockbroker, helped him consolidate power by squeezing Congress loyalists out of non-state institutions, such as the state's banking and dairy cooperatives, which are economically powerful and influence the lives of millions of voters.
Modi is the longest-serving chief minister in Gujarat's history, a fact that has helped lure high profile defectors away from Congress ranks, another favoured tactic. In the last two years, hundreds of Congress workers have switched sides in the state, including some of its top leaders.
While Modi may not press legal cases against the Gandhis or their associates for alleged corruption - a move some in his party would encourage - his resounding victory helps him make the BJP India's natural party of power, not Congress.
"We're looking at being out of power for 10 years," said the party strategist.