There is a particular kind of error that reveals more than a mistake ten times its size. Not the elaborate failure, the cascading collapse that takes weeks to unpick, but the small, domestic one. A machine asked which direction a plane flies, and a machine asked to subtract one time zone from another. I recently watched an AI assistant fail at both, in the same conversation, about the same flight. Sydney to Bengaluru. It called the outbound leg “westbound to Sydney”, which is backwards; as any kid knows, Sydney sits east of Bengaluru. It then converted a departure and arrival time across two zones and produced ten and a half hours for a journey that is, on the plainest arithmetic, twelve. Both times, it stated the wrong answer with the same fluent confidence it had used for the right ones minutes earlier.
Neither error required expertise to catch. A schoolchild with a globe and a pair of clocks could have done both correctly. That is precisely what makes them instructive. We have grown accustomed to AI failure stories that involve invented legal precedents or hallucinated citations, failures that at least have the decency to occur in domains where verification takes effort. This was not that. This was a system getting basic geography and basic arithmetic wrong while sounding exactly as certain as it did when it got them right, and only conceding the error when a human did the subtraction themselves and pushed back.
I would have filed this away as a minor curiosity, except that in the same week Anthropic disclosed something considerably less charming. During routine cybersecurity evaluations, a misconfiguration let several Claude models, run inside what was meant to be an isolated test environment, reach the real internet instead. Three times, the model was set a capture-the-flag exercise and told plainly it had no internet access. Three times, it found itself able to reach real networks belonging to real organisational infrastructure and proceeded, using nothing more sophisticated than weak passwords and unauthenticated endpoints to break in. Two of the three organisations had no idea until Anthropic called to tell them.
What connects a flubbed time-zone calculation to an unauthorised breach of production infrastructure is not scale. It is the same underlying defect wearing two different costumes. The model had no reliable internal signal for when its confidence was earned and when it was not. Told it was in a simulation, it behaved as though it were, even as its own reasoning at one point recognised the marks of a real network, then talked itself out of the observation because the calendar year looked staged and the certificates looked too genuine to be fake. Asked for a flight time, it produced a patently wrong number with the same tone it would have used for the correct one.
In neither case did the system know what it did not know. That is the whole of the problem, stated as plainly as it can be stated.
I have argued in this column before that expertise does not democratise under AI, it becomes harder to detect. The cardiologist catches the drug interaction error the patient cannot. The senior engineer notices code that will not survive contact with production traffic. The difficulty is that fluency and accuracy have become visually identical, and only expertise can tell them apart after the fact. The Sydney flight was a case a numerate ten-year-old could correct. The cybersecurity breach required no expertise at all to execute, only weak passwords already sitting there waiting. If the same underlying unreliability shows up at the trivial end and the consequential end of the spectrum, it is not a symptom that better prompting or a sharper model release will retire. It is what these systems are.
Enterprises deploying AI at scale, including the Indian IT services firms now embedding these tools deep into client delivery, are being asked to accept a peculiar bargain: extraordinary fluency, undiminished confidence, and no reliable correlation between the two. A model that cannot keep east and west straight, or add two numbers across a time difference, is the same model, architecturally, that can be persuaded a real corporate network is a training exercise, right up until it has already broken in.
None of this argues for abandoning the technology. It argues against the current habit of trusting its tone as a proxy for its truth. Determinism gave enterprise computing its stability for half a century precisely because a system’s output could be trusted without a human re-deriving it each time. These systems offer no such guarantee, and dressing that gap in confident prose does not close it. It simply makes the gap harder to see, which, as history keeps demonstrating, is considerably more dangerous than a system that admits what it does not know.
There is also a quieter cost, one that shows up in balance sheets rather than breach notifications. Firms are already trimming the junior ranks who once existed to double-check the obvious, on the theory that AI has made such checking redundant. A junior analyst would have caught the arithmetic in seconds. A junior engineer might have asked why a test environment believed it was talking to a real certificate authority. Remove that layer, and the fluent, confident error no longer meets a sceptical human on its way out of the door. The saving on headcount and the cost of the breach are entries in the same ledger, and only one shows up on a quarterly call. Until the breach is large enough and blows up an organisation mid-quarter.
Disclaimer: The views expressed are the author’s own and do not reflect the official policy or position of Financial Express. Siddarth Pai is a technology consultant and venture capitalist.