The ceiling problem

There is a foundational bet being placed, quietly but with enormous sums of money, at the heart of the artificial intelligence (AI) industry. The bet is that large language models, arranged in sufficiently clever configurations, will eventually become capable of doing anything a human knowledge worker can do. The autonomous agent. The tireless, infinitely scalable mind that never sleeps, never complains, and never asks for equity.

It is a seductive idea. It is also running into mathematics.

Imagine an AI agent reviewing thousands of contracts. For months, everything appears to work. The outputs are coherent, the recommendations sensible, the productivity gains measurable. Then the system encounters a situation whose complexity exceeds its ability to reason through it. It does not stop and ask for help. It does not announce uncertainty.

It produces an answer that looks correct and moves on. The organisation consuming the output often does the same.

This is the problem that Vishal Sikka and his son Varin Sikka examine in their paper, “Hallucination Stations: On Some Basic Limitations of Transformer-Based Language Models.” Their argument is that LLMs may face a structural ceiling on their ability to carry out agentic tasks beyond a certain complexity threshold: one that scaling and engineering can raise, but not ultimately eliminate.

The mechanism is not mystical. Every transformer model, which means essentially every GenAI product today, performs a finite amount of computation for each response. When a task requires more computation than the architecture can provide, the model does not recognise the shortfall. It hallucinates.

The output may be fluent, authoritative, and perfectly formatted, yet wrong in ways that are difficult to detect. The hallucination, in this view, is not merely a bug. It is the system colliding with a limit it cannot acknowledge.

The theoretical foundation reaches back to a theorem established by Juris Hartmanis and Richard E Stearns in 1965. Its central insight is simple: certain problems possess an irreducible computational cost. No amount of cleverness allows a thousand-piece jigsaw puzzle to be solved in the time required for a ten-piece one.

The Sikkas apply this logic to transformers. When verification complexity exceeds the model’s computational capacity, it cannot reliably complete or audit its own work. Equally important, it cannot recognise that it has failed.

The industry response has been predictable. Scaling will fix it. Chains of agents checking one another’s work will fix it. Retrieval augmentation will fix it. These interventions are real and often useful. They move the ceiling upward. They do not remove it.

Organisations are already commissioning AI agents for increasingly complex tasks. Yet the complexity of verifying those tasks is itself substantial. Asking an AI system to perform such work and verify its own output is roughly equivalent to asking a ruler to calibrate itself. The instrument and the source of error are one and the same.

The deeper problem is organisational. AI-generated output passes through layers of human review conducted by people increasingly unequipped to identify subtle mistakes. AI output often appears most authoritative to those least capable of auditing it. Not only does the system fail to recognise its own errors; many of the humans downstream cannot recognise them either.

The organisation gains speed while simultaneously losing its capacity for correction. Applications operating near the complexity threshold described by the Sikkas may produce work that appears complete while quietly concealing unresolved problems behind confident prose.

The practical path forward lies in hybrid architectures. Systems that combine GenAI with formal verification engines, deterministic software, domain-specific tools, or expert human checkpoints can address classes of problems that a standalone model cannot. The model becomes one component of a larger system rather than the system itself. This approach is architecturally unglamorous. It is also likely to be correct.

The obvious engineering rebuttal deserves attention. Silicon Valley is not building pure language models; it is building systems. When a model struggles with a mathematical problem, engineers give it access to a Python environment.

The model writes code, the code executes deterministically, and the result returns. Wrapped inside conventional software, the model appears to bypass the limitation.

This is an important point. But it relocates the problem rather than resolving it. The decision about when to delegate, what to delegate, and how to frame the problem occurs inside the transformer before the external tooling is invoked.

A model that cannot recognise it has reached its own ceiling may not know when to reach for the calculator. The hallucination can occur before the scaffolding engages. Better scaffolding raises the threshold. It does not eliminate it.

The theorem the Sikkas invoke is older than many of the engineers building the systems they critique. If they are right, the ceiling has always existed. The industry has simply been advancing quickly enough that it has not yet encountered the limit in ways that cannot be explained away.

The more important question is whether organisations deploying AI agents understand the risk well enough to build responsibly, rather than discovering, too late, that they have automated beyond their capacity to verify.

The real risk is not that AI agents make mistakes. It is that organisations deploying them may gradually lose their ability to recognise those mistakes when they occur.