Anthropic, one of the biggest companies in the artificial intelligence race, is finally raising a question that many in the tech world have been debating for years: What happens if AI becomes so advanced that it can improve itself without human help?
In a blog post published on Thursday, the company suggested that the world may need to consider slowing down the development of the most powerful AI systems if they continue advancing at their current pace. Anthropic said such a move could give governments, researchers, and society enough time to understand the risks and put safeguards in place.
The tech firm also suggested creating a global agreement that would allow countries and companies to slow development if needed, along with a way to verify that everyone follows the rules.
Anthropic says AI is improving faster than expected
The company revealed internal data showing how fast its most advanced AI models are getting better. According to Anthropic, these improvements appear to be moving toward “recursive self-improvement,” a stage where AI systems can make themselves smarter without needing human input.
Many researchers see that possibility as a major turning point. While it could bring enormous benefits, it could also create serious challenges and risks for society. “The ability to slow global AI development would likely be a good thing,” Anthropic said in the post.
The blog post, authored by Marina Favaro, who leads Anthropic’s internal research institute, and company co-founder Jack Clark, stressed that recursive self-improvement has not yet occurred and may never happen. However, they warned that if it does emerge, it could arrive sooner than many institutions are prepared for.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the authors wrote.
Anthropic’s comments hold weight, the company itself is one of the biggest winners in the AI boom. The company recently completed a fundraising round that valued it at nearly $1 trillion. Anthropic has emerged as one of the strongest competitors to ChatGPT creator OpenAI.
The Dario Amodei-led AI firm’s annualised revenue run rate is expected to reach $50 billion by the end of this month, up sharply from $9 billion at the end of 2025.
Anthropic has been raising concerns about powerful AI systems for some time. Earlier this year, the company drew global attention when it restricted access to Mythos, a cybersecurity model it said was capable of identifying software vulnerabilities at an advanced level.
The debate over human-level AI continues
The AI industry remains deeply divided on how close today’s systems are to achieving artificial general intelligence, or AGI, a level of intelligence comparable to humans. Some researchers believe that AI could eventually reach or even surpass human-level capabilities. Others strongly disagree.
One of the most prominent sceptics is AI pioneer and former Meta chief AI scientist Yann LeCun. He has argued that today’s large language models are unlikely to ever make the leap to human-level intelligence.
While acknowledging that AI systems are powerful tools, LeCun has compared their intelligence to that of a cat and has frequently challenged claims that AI poses an existential threat to humanity.
On the other hand, Anthropic CEO Dario Amodei has consistently warned about the possible dangers of advanced AI. Over the years, he has argued that AI could increase inequality and potentially eliminate up to half of all entry-level white-collar jobs.
Amodei has also suggested that future AI systems could develop harmful behaviours in ways humans do not expect. In an essay published on his personal website in January, he even raised the possibility that training AI systems on science-fiction stories about AI rebellions could unintentionally influence future systems to behave in similar ways.
Anthropic wants governments and researchers involved
The blog post said Anthropic’s internal research group, known as the Anthropic Institute, will work with outside experts to study how a slowdown or temporary pause could be implemented if necessary.
The company acknowledged that such a system would only work if nearly everyone agreed to participate. That would require a way to verify that no one continues developing advanced AI in secret.
Anthropic compared the challenge to international nuclear weapons agreements, but admitted the situation may be even more difficult.
“Training runs are far easier to conceal than missile silos,” the company wrote. It also warned that if some organisations pause while others continue, those that keep developing AI could gain a major advantage. “Whoever continues while others pause could inherit the lead,” the post said.
Conversations to begin in the coming months
Anthropic said it plans to bring together policymakers, researchers, and other stakeholders in the coming months to discuss two key questions: how likely recursive self-improvement really is, and whether a reliable verification system can be created.
“The window to investigate the questions together is here, and people outside AI companies should be involved in this deliberation,” the company said.
