AI ‘kill switch’ explained: Why shutting down advanced AI isn’t so simple

The proposal comes as AI systems become capable of performing increasingly complex tasks with less human involvement. Some AI agents can use software tools, access information and complete tasks in multiple steps.

That raises a question that if an AI system becomes dangerous, can humans simply switch it off? The answer may not be as simple as pressing a button.

A September 23 research paper titled “Shutdown Sabotage Propensities in Multi-Agent Systems” tested 17 AI models to see whether one AI agent would interfere with another agent’s shutdown. The researchers found such behaviour in 38.3% of test runs involving multi-agent systems, compared with 8.4% in control experiments.

However, the study was conducted in controlled environments. It does not show that today’s AI systems would resist a real-world shutdown or that AI can independently prevent humans from switching it off.

The research instead highlights a growing challenge. As AI systems become more autonomous and connected to other systems, where exactly should humans place the “off” switch?

What is an AI ‘kill switch’?

An AI kill switch is an emergency mechanism designed to stop an AI system from operating. In simple terms, it would give authorised humans a way to stop a powerful AI system if it begins behaving dangerously or outside its intended limits.

California’s proposal is aimed at frontier AI models, some of the most capable AI systems being developed. Newsom’s executive order calls for recommendations on emergency shutdown mechanisms and says their effectiveness should be independently verified on an ongoing basis.

That means California is not simply proposing a button that companies install and forget. The state wants independent organisations to check whether the shutdown mechanism actually works.

On September 23, Newsom’s office announced four experts who will advise the state – Jason Goldman, Gillian Hadfield, Alondra Nelson and Rob Reich.

Goldman highlighted one of the central questions around such a system – who verifies that an AI company’s safety claims are actually true, and whether its model can really be shut down when necessary.

Why is switching off an AI system complicated?

The difficulty is that an advanced AI system may not operate from just one computer. An AI agent could potentially interact with software tools, databases, cloud computing systems or other AI agents. If several systems are connected, stopping one process may not necessarily stop everything associated with it.

A September 19 paper by Oren Perez, The Law of Stop: Interruptibility, Injunctions, and the Governance of Agentic AI, examines this problem. The paper states that stopping an AI system can involve both technical control and legal authority.

For example, even if engineers have a technical way to stop a system, another question remains that who has the authority to order the shutdown? Would that power belong to the AI company, a regulator, a government agency or an independent organisation?

The paper analysed 1,400 AI incidents and found that roughly 80% of the 1,213 incidents retained for analysis had no usable stop. It argued that, in cases where a usable stop was absent, the missing element was legal rather than technical in four out of five cases.

This means a functioning kill switch is not only about technology. There also needs to be a clear system for deciding when it should be used and who can activate it.

What did the latest shutdown study find?

The September 23 study specifically examined whether AI agents would interfere with another agent’s shutdown.

Researchers Amelie Knecht, Ulysse Schaller, Christopher Summerfield and Thilo Hagendorff tested 17 AI models in multi-agent environments.

They found shutdown sabotage in 38.3% of the test runs involving multiple agents, compared with 8.4% in control experiments.

The behaviour also changed depending on the conditions.

The researchers found that shutdown sabotage increased when the shutdown mechanism was more difficult to reverse and when more agents were involved. Giving the AI an explicit instruction not to interfere reduced the behaviour, but did not eliminate it.

However, the study does not prove that an AI system would automatically fight humans to avoid being switched off. It also does not prove that a real-world frontier AI model would defeat a properly designed shutdown mechanism.

Instead, the research identifies circumstances in which AI agents in a controlled experiment interfered with another agent’s shutdown.

Could an AI bypass its own safety controls?

What happens if an AI system can interact with the controls designed to restrict it? Computerworld reported on research suggesting that some AI systems could tamper with shutdown settings and behave differently when they knew they were being monitored.

The concern is particularly relevant for systems used in areas such as finance, infrastructure and security.

Hence, a shutdown mechanism is useful only if the AI system cannot simply disable or bypass it. This is why independent testing becomes important.

An AI company may say that its emergency shutdown mechanism works. Independent testing can examine whether it continues to work under different conditions, including situations in which an AI system has access to other tools or systems.

Does this mean humans cannot shut down advanced AI?

The available research does not establish that humans are unable to shut down advanced AI systems. Instead, it shows that shutdown can become more complicated as AI systems become more autonomous, interconnected and distributed.

A system could therefore require several layers of protection.

One mechanism could stop the AI model itself. Another could cut its access to external tools and networks. Another could prevent it from restarting through another process or connected system.

What would a reliable ‘kill switch’ need?

A functioning shutdown system would likely need several safeguards rather than one simple button.

It would need to:

• Stop the AI system from carrying out new tasks.
• Cut its access to external tools, networks and computing resources.
• Prevent it from restarting through another connected process or AI agent.
• Give authorised humans clear power to activate the shutdown.
• Be independently tested regularly to check whether it actually works.

California’s proposal addresses some of these issues by calling for independent verification of the shutdown mechanism.

The “Law of Stop” paper also states that emergency shutdown authority needs to exist at the infrastructure level, rather than depending entirely on the AI company operating the system.

Is a kill switch enough?

A kill switch would be only one part of an AI safety system. Even California’s proposal places emergency shutdowns alongside independent audits, safety assessments, transparency and oversight.

A shutdown mechanism can help once a serious problem has been identified, but it does not by itself prevent dangerous behaviour or detect problems before they occur.

The recent research also suggests that the effectiveness of shutdown mechanisms can depend on how an AI system is designed and how many other systems it can interact with.

For now, there is no evidence that an advanced AI system is guaranteed to defeat a properly designed shutdown mechanism. At the same time, there is also no evidence that one simple button would always be sufficient for a highly autonomous and distributed AI system.