Who will put on the brakes first?

But what Jacob Coxon, a former Anthropic and OpenAI AI researcher, said recently should make anyone who has been swept up in that excitement sit up.

Coxon is no outsider waxing eloquent about the harms of AI. He has worked on AI pretraining at both companies and recently left Anthropic, reportedly giving up equity that had not yet vested. His warning was stark: the companies building the most advanced AI systems are racing towards self-improving superintelligence without having solved how to keep such systems reliably under human control. He has said people working on the technology themselves believe there is a possibility of AI causing catastrophic, even existential, harm within this decade.

For an industry that has spent years selling the promise of increasingly capable machines, such a warning is bound to disturb some of the optimism around AI.

Now, according to reports, OpenAI CEO Sam Altman has said the company is open to slowing the development of its AI systems amid growing concerns over the safety of increasingly capable models. And the risks are no longer entirely hypothetical. According to one report, Anthropic said it had disrupted attempts to use its Claude models to support biological weapons research, develop software for missiles and armed drones, and carry out a suspected Russia-linked cyber espionage campaign against the Ukrainian government and military targets. It also said Chinese AI companies had attempted to extract capabilities from its models.

So, the companies building frontier AI are beginning to acknowledge that the technology may be moving faster than the safeguards around it. But recognising the problem does not solve it.

The bigger question is who slows down first. If OpenAI decides that its next model needs another six months of safety testing while Anthropic or Google continues at full speed, OpenAI could lose its technological lead. If US companies slow down while companies elsewhere continue developing more capable systems, economic competitiveness and national security enter the picture. This is the central problem facing frontier AI. Every company may prefer a world in which the industry develops powerful systems more cautiously. But no company wants to be the only one to do so.

There is a familiar historical parallel. During the Cold War, the US and Soviet Union both understood the risks of an unrestricted nuclear arms race. Neither, however, could afford to assume that restraint would be reciprocated. The eventual response was not simply to tell one side to stop, but to create mechanisms through which both sides could place limits on the race.

AI is obviously not a nuclear weapon. But the competitive logic has a parallel — everyone may benefit if the race is slowed, while each participant has an incentive to keep running.

That is why voluntary restraint has limits. It also explains the growing calls for government intervention. Anthropic CEO Dario Amodei has already spoken about mandatory testing of advanced AI systems, independent evaluation, and government powers to intervene when systems pose unacceptable risks.

There is a strong case for this approach. If the same rules apply to every frontier developer, safety ceases to be a competitive disadvantage. OpenAI does not have to worry that delaying a risky system will hand an advantage to Anthropic, and vice versa.

But regulation creates another problem. Governments are being asked to regulate a technology evolving at a pace traditional policymaking is not designed for. For instance, aircraft can be tested against known engineering standards. Medicines go through established clinical and safety procedures. AI is different. A model can acquire new capabilities as it is scaled. A regulator may be trying to certify something whose behaviour is still changing.

That does not mean governments should wait until they understand everything about AI. Logically, regulation should focus less on prescribing the technology and more on defining acceptable levels of risk. Automobile provides a useful parallel. Governments did not stop developing more powerful cars because they could be dangerous. They introduced driving licences, speed limits, safety standards, and traffic rules. A similar approach could work for AI. Less capable systems could face lighter requirements, while increasingly autonomous systems face progressively tougher testing and disclosure obligations.

However, the difficult part will be deciding where those thresholds lie. Governments will need technical expertise independent of the companies they regulate, access to sufficient information from frontier labs, credible third-party testing, and the ability to revise rules as capabilities change.

There is also a danger that excessively expensive or complicated compliance could strengthen the biggest AI companies by making it harder for smaller competitors to enter the market. So, regulation should set safety conditions, not dictate the pace of innovation. That would perhaps address the industry’s biggest incentive problem and ensure that the company that applies the brakes first is not simply the one that gets left behind. But again, it is difficult to say whether this would work either.