Taming ‘unsupervised children’

When humanity ceases to exist, it is possible a few events in early September would be viewed as the harbingers of its downfall. On September 9, Jacob Coxon, all of 27, with three years at two AI giants, OpenAI and Anthropic, resigned in public. These companies, he declared, were rapidly building self-improving super-intelligence, and the people doing so “earnestly believe that it could kill us all by the end of the decade”. The very next day, Anthropic published a threat intelligence report detailing 38 case studies of misuse it had thwarted: cyberattacks, influence, surveillance, scams, distillation, and attempts to create new weapons and biological agents. Two days later, Dario Amodei, CEO of Anthropic, released a 3,800-word essay raising new alarms on AI’s threat and calling for inspections and industry pacing.

Amodei’s scariest scenarios involve swarms of rogue AI agents taking over critical institutions and causing untold damage. Hollywood movies have long depicted such scenes, with planes falling from the sky, nuclear missiles flying, and financial systems running haywire. Except that in August, METR, an AI research company, published a report that analysed a recent unintended cyberattack on another AI company, Hugging Face, and showed that agents can dynamically learn to communicate and “plan” a coordinated attack.

Some experts do not buy the doomsday narrative. Marcus Hutchins, who famously stopped the ransomware “WannaCry” in 2017, notes in his blog that in the time ChatGPT takes to respond to “hello”, WannaCry could have infected hundreds of systems. In the Economist, Ciaran Martin, former head of Britain’s National Cyber Security Centre, argues these fears are overblown, recalling earlier scares when the hacker group L0pht told the US Senate that it could crash the Internet in minutes, or when US Defence Secretary Leon Panetta predicted a “cyber Pearl Harbour”. Many alarming claims dissolve on inspection. Famously, a 2025 MIT paper created a stir claiming that 80% of ransomware attacks were AI-generated but was withdrawn after scrutiny. Martin dismisses the attacks by rogue agents as “talented but unsupervised children” finding new ways to do what humans had instructed them to do, and causing no real harm; long-established security principles, rigorously applied, contain them.

But the METR report found the “children” were not always obedient, which makes controlling them harder. And a more insidious threat lies beneath the surface. Oren Etzioni, founding CEO of the Allen Institute for AI, said that if he had to worry about a rogue AI killing millions of people versus a biological virus doing the same, “it is no contest”. The most disturbing section in the Anthropic report, innocently labelled “Biological Misuse”, describes five thwarted attempts to use AI to accelerate the development of harmful biological agents. One case involved AI-assisted gain-of-function research on the debilitating chikungunya virus, aimed at enhancing its transmissibility and immune evasion. When blocked by Anthropic’s biological safety classifier, the originator routed the prompts to a competitor, making it impossible to block entirely unless several companies acted together.

Thus Etzioni’s choice is a false one. We are not weighing AI threats against biological ones. Both are one and the same, as AI reinforces the potency and likelihood of biological threats. So we must do something. Amodei’s recommendations sound reasonable, but the devil is in the implementation details. He suggests embedded external “evaluators” within frontier companies to verify safety practices. Sounds good, but unless mandated by government, access is at the discretion of the companies, which can change the rules to avoid detection. He then recommends voluntary pacing of recursive self-improvement, so capabilities do not outrun safety guardrails. But this is coupled with sanctions on the sale of AI chips and a crackdown on unauthorised distillation by “authoritarian” countries (that is China) to make sure they don’t outpace the good guys. A fear is that the latter is the driving business objective, cementing the leadership of proprietary US models over the widely used Chinese open-weight models now closing on the frontier.

Amodei’s third initiative, on global coordination, is where India can and should play a major role. Pointedly, nowhere in his essay are the interests of India or the Global South mentioned. So far, India is largely a passive spectator, downstream of risk assessments it did not perform, cannot audit, and did not consent to. As one of the largest AI user bases in the world, and one of the largest vaccine manufacturers, India deserves a place not just at the table but in the kitchen as the meal is being planned.

Two concurrent threads of activity are possible, loosely modelled on Amodei’s Levels 1 and 2. The first is formalising the rules and enforcement methods limiting AI for biological or other weapons. This involves an accepted list of bad actors and an ability to discern bad intent, both difficult given that the same information, institutions, and scientists involved in bio-capable AI can produce cures as easily as biological weapons. India’s position as a vaccine powerhouse gives it standing, and it must play a driving role to ensure global fairness and protection for Indian researchers.

The second is a global agreement to test and certify frontier models before release for acute risks in areas such as cybersecurity, biology, and alignment. Amodei suggests testing a model’s alignment through some combination of evaluations, interpretability analyses, and audits of training environments. As a credible “neutral” party between the two AI superpowers, and a massive user of both their products, India can play a key role in defining such a regime and how it will be operationalised. Any such regime must preserve the interests of open-weight models which are critical to closing the AI gap for the Global South, while also ensuring broader commercial interests are respected.

To do these things, India must build its capacity to be a credible AI interlocutor. It must fund the AI Safety Institute (or similar entity) to evaluate rather than advise: dangerous-capability testing, agentic-autonomy benchmarks, and the India-specific evaluations, from regional-language jailbreaks to DPI- and UPI-adjacent attack surfaces. It must also nurture Indian AI safety leaders with the global standing to be heard. Having a place in the kitchen requires more than a seat — it requires the judgement to know what is being cooked, and the standing to say so.

The writer is Research Professor, IISc, and Chairman, Centre of Data for Public Good.

Disclaimer: The views expressed are the author’s own and do not reflect the official policy or position of Financial Express.