Supreme Court warns against AI ‘hallucinations’ after NCLT relied on fake precedents

The verdict came in an insolvency matter involving Pooja Ramesh Singh, Jammu and Kashmir Bank Ltd and Essel Infraprojects Ltd. While examining the NCLT’s ruling, the apex court found that several authorities cited to support the tribunal’s reasoning were either entirely fictitious or incorrectly attributed to genuine judgments.

‘Zero tolerance’ for fake AI-generated precedents

The bench comprising Justices PS Narasimha and Alok Aradhe said, “The production of fake, non-existent, and hallucinated material and its utilisation as precedents in law, is like the release of methyl Isocyanate in the province of law and justice: invisible, insidious, and catastrophic by the time anyone notices. It not only contaminates but takes away the very lifeblood of judicial determination.”

“It is necessary for Courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification. It is a misconduct on the part of an advocate to cite such judgments without verification,” the verdict said.

The bench said it was equally a “serious lapse” if a judge relies on such a fake or hallucinated AI-generated material as precedents in support of the determination.

“We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication,” it said.

The judges underlined that the integrity of judicial reasoning cannot be compromised by unverified AI-generated content, regardless of how it enters the decision-making process.

BCI asked to frame safeguards on AI use

The court clarified that its observations should not be interpreted as opposition to artificial intelligence itself, but rather as a warning against presenting fabricated material as authentic legal precedent.

“And we reiterate and declare zero tolerance for the Bar as well as the Bench to cite, refer to, or rely on such material. It is also clarified that our judgment shall have no bearing on the rightful use of AI, but on the presentation or reliance on fake or hallucinated material as if it were a court precedent,” it said.

Recognising the need for institutional safeguards, the bench directed the Bar Council of India (BCI) to examine the issue and recommend standards governing the use of AI-generated material in legal proceedings.

“So far as the responsibility of the bar is concerned, we direct the Bar Council of India (BCI), being the apex statutory body, to constitute a committee and deliberate on this issue of members of the bar submitting such fake and hallucinated material before the Court as if they are precedents of law,” it said.

The verdict said the apex bar body must take up this issue with utmost seriousness, deliberate earnestly, and prescribe a guiding principle to prevent such occurrences, along with the disciplinary action that will follow a violation of the norms.

During the proceedings, Jammu and Kashmir Bank informed the court that its counsel had not cited the fabricated cases and that the NCLT had relied on them through its own research. However, the Supreme Court held that the origin of the error did not diminish its impact on the justice system.

“More than the inevitable consequence of setting aside such judgments, what is significant for our decision-making is our resolve to adopt AI technology in aid of adjudication, while at the same time asserting and declaring total and absolute control over adjudication, with a human in the loop at every stage,” the bench observed.