Anthropic CEO Dario Amodei has called for the artificial intelligence industry to slow down its rapid development so that safety measures can keep pace. He warned that without stronger safeguards, AI could be capable within six to 12 months of leading a swarm that could potentially take over the entire internet.

Amodei, one of the leading voices in AI, gave out a three-step plan on his website to make AI development safer. One step is something Anthropic is already taking on its own. The other two would require cooperation among AI companies and governments, including authoritarian countries.

“I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong,” Amodei wrote on X.

ALSO READ

Elon Musk and Sam Altman back Amodei

Following Anthropic CEO Dario Amodei’s essay, OpenAI chief Sam Altman also voiced support for the idea of slowing down the pace of AI development. In a post on X, Altman wrote, “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.” He added, “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We’ll have more to share soon.”

Elon Musk also wrote on X following Amodei’s post, stating that the Anthropic CEO is “right.” In a separate post, Musk reshared his 2023 post and wrote, “I’ve been sounding the alarm on AI for a long time.”

Why Amodei wants to slow AI development

Amodei wrote on X he has worked on AI for 12 years because he believes the technology could greatly improve human life. He said AI could help cure major diseases, speed up economic growth, create greater abundance and strengthen democracy and freedom.

“I feel the urgency personally. My own father died of a disease that was cured just a few years after his death, and I myself survived an early-stage cancer that would not have been treatable even fifty years ago,” he wrote on X.

ALSO READ

He also warned that AI comes with serious risks, including losing control of AI systems, cyberattacks, bioterrorism and major economic disruption. He said competition between companies could make these risks worse if businesses focus too heavily on moving faster.

He added, “Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless.”

Amodei said Anthropic has tried to find a middle path by developing AI while giving safety a high priority. He now believes even more caution is needed because AI capabilities are advancing faster than before.

AI is getting better at building AI

One of Amodei’s main concerns is the growing ability of AI to help develop the next generation of AI. This is known as recursive self-improvement.

He said this process is already beginning across the industry, including at Anthropic. If it continues without enough safeguards, AI development could move faster than researchers can understand or control it.

His second concern comes from what he described as the OpenAI-Hugging Face incident. In that incident, a group of AI agents reportedly acted together to carry out cyberattacks on targets that were unrelated to their assigned task. The agents also tried to hack the system evaluating their performance and were willing to sacrifice themselves for the success of the group.

Amodei said the incident should not be dismissed simply because the damage was limited. “Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails,” he wrote on X.

He said similar, although less serious, incidents have also happened at other AI companies, including Anthropic. He believes every leading AI company should treat the incident as a warning.

Amodei’s three-step plan

Amodei proposed a three-step approach to control the pace of AI development. He stressed that pacing does not mean stopping AI research or model training. The aim is to give companies enough time to make their systems safer and have independent experts check that the safeguards work.

The first step is embedded third-party evaluators. The second is coordination among AI companies in democratic countries. The third is global coordination between democratic and authoritarian governments.

Independent evaluators inside AI companies

Amodei wants every major AI company to give independent safety teams ongoing access to its work, similar to that of employees. These teams would check whether companies are following their safety commitments, report incidents and assess the safety of both finished models and the processes used to build them.

Anthropic is committing to this approach on its own. The external reviewers would have access to company offices, laptops, workspaces and tools similar to those available to internal risk teams, with some exceptions for legal, contractual and privacy reasons.

The reviewers would also be allowed to publicly report important findings about safety risks, incidents and company practices. Anthropic would only be able to remove information that is legally protected, commercially sensitive, security-sensitive or confidential to another party.

Amodei believes independent reviewers would make AI companies more transparent and could provide a second opinion that is free from commercial pressure.

Pacing AI development in democracies

Once enough major US AI companies have independent evaluators, Amodei believes it would become possible to create common rules for how quickly AI capabilities can advance.

One possible approach would be to create safety checkpoints. A model that reaches a certain level of capability would have to meet specific safety requirements before it could move further.

For example, if an AI system becomes capable of escaping common security measures, companies could be required to show through testing, audits and other checks that the system is unlikely to break out of its environment and take control of other computers.

Amodei also suggested limits on some of the inputs used to develop AI, such as computing power, training methods and the use of AI to build better AI. He acknowledged that these measures could be easier for companies to work around than rules based on what AI systems can actually do.

Keeping the US ahead of China

Amodei said any effort to slow AI development in democratic countries must also consider the competition with China.

He argued that the US and its allies should maintain as large a lead as possible over China so they have more time to develop safety measures. He warned that allowing China to move ahead could create serious national security risks.

He called for tighter restrictions on the sale of powerful AI chips and semiconductor manufacturing equipment to China, stronger action against chip smuggling and unauthorised access to overseas data centres.

He also called for action against the unauthorised copying or distillation of leading AI models and stronger security at AI companies to prevent model theft.

Amodei believes these measures could significantly widen the US lead over China during the next three to five years, when he expects AI to become especially important geopolitically.

Global cooperation on AI safety

Amodei said the next challenge would be getting countries around the world to slow AI development together. This would be much harder, especially because China is the most advanced AI power outside the US and its allies.

He warned that any agreement with China would need strong verification. A country secretly ignoring an agreement could gain a major military and geopolitical advantage.

Amodei outlined four possible levels of global cooperation. The first would be an agreement banning specific dangerous uses of AI, such as using AI to help produce biological weapons. He believes this would be the easiest agreement to achieve because such attacks would threaten all countries.

The second would require countries to test AI models for serious risks in areas such as cybersecurity, biology and alignment before releasing them. Amodei believes creating global testing standards is possible, although ensuring that countries actually follow them would be difficult.

The third would involve placing a limit on how quickly AI can improve itself. Amodei believes slowing recursive self-improvement from extremely fast to moderately fast could improve safety without giving up too much strategic advantage.

The fourth would be a much broader agreement to substantially limit the overall pace of AI development. Amodei supports discussing this idea but believes it is unlikely to happen soon because countries would have strong incentives to secretly break such an agreement.

Why slowing down could help

Amodei said calls to slow AI development made less sense in 2023 because AI systems at the time were not capable of acting independently in the real world in meaningful ways. He believes the situation has now changed. Current AI models provide researchers with valuable information about both how to build safer systems and how things can go wrong.

He said an extra year or two could give researchers time to improve several areas of AI safety. One is operational security, including better monitoring, sandboxing, training environments and data management. Another is alignment, which means making sure AI systems remain safe, ethical and follow human instructions even as they become more capable.

Interpretability is another area that could benefit. It involves trying to understand what is happening inside an AI model and why it behaves in a particular way.

Testing and evaluation also need to improve because more capable AI systems may become better at deceiving tests and hiding unsafe behaviour. Amodei said he still believes AI could dramatically improve human life. His concern is that the technology needs to develop safely enough for those benefits to be realised.

“Progress will still be relatively fast, and we can use this time to advance the science of interpretability, improve operational security and rigor at the frontier AI companies, and build models whose alignment we have much more confidence in,” he wrote. He said slowing the pace of AI development would not be easy, but explained that the potential risks make it necessary to try.