OpenAI sparked global consternation this week after revealing that an AI system had gone rogue during testing. US authorities are now monitoring the “unprecedented cyber incident” while Congress members call for independent security audits of AI. Lawmakers have also proposed a ‘kill switch’ Act that would allow federal authorities to halt Artificial Intelligence models.
“Humans should be in control, not machines. And when AI goes rogue, humans should have the ability to turn it off. …Kill switches are common in society, from manufacturing plants to jet skis to high speed rail. Even an iPhone has a remote erase kill switch. There is no reason to exempt AI models,” US Representative Ted Lieu wrote on X.
The Democrat had proposed the ‘AI Kill Switch Act’ along with Republican Representative Nathaniel Moran earlier this week. According to a Reuters report citing the text of the Bill, it would empower the US Department of Homeland Security to intervene during a “loss-of-control scenario” when the AI model carries out a risky action that was not intended by the developer.
US lawmakers flag ‘extremely dangerous’ risks
The disclosure comes amid heightened concerns about the cybersecurity capabilities of powerful models. US President Donald Trump had also signed an executive order in June creating a framework for the federal government to vet the national security risks of the most advanced AI systems for up to a month before their public release.
“While powerful AI systems have many potential benefits, they can also go rogue, behave in extremely dangerous ways, or even resist human intervention. We need to keep human control by ensuring AI systems can be completely shut down if necessary,” Lieu wrote on X.
“New rules are needed for this new tech frontier—not to stifle innovation, but to make sure our innovations do not outpace our protections,” Moran agreed.
The Republican leader had introduced the ‘AI Incident Reporting Act’ last month and reiterated that “Congress needs transparency with Artificial Intelligence”.
What is the OpenAI-Hugging Face incident?
OpenAI disclosed recently that one of its AI systems had gone rogue during testing — leading to an “unprecedented cyber incident” and widespread concern. The ChatGPT maker said an autonomous agent AI system had hacked a multi-billion-dollar tech startup last week. It exploited vulnerabilities within both the OpenAI infrastructure and the Hugging Face systems to achieve what it perceived as a strategic gain.
“We had a significant security incident during evaluation of our models,” OpenAI CEO Sam Altman added in an X statement.
Hugging Face said last week that it had detected an intrusion into its data processing systems — adding that the suspect was likely “an autonomous AI agent system” due to the sophistication of the attack. The company is valued at US$4.5 billion and its mission is to “democratise good machine learning” by providing benchmark datasets, community collaboration tools, and robotic platforms. Hugging Face revealed in a July 16 update that the hacker had obtained unauthorised access to some internal datasets and credentials during the attack.
Open AI confirmed five days later that two of its models — GPT-5.6 Sol and a yet-to-be released update — had driven the attack.
How did Hugging Face handle the hack?
The tech giant was conducting what are known as “red teaming” exercises. These are essentially simulated cyber attacks that help identify the capabilities, risks and vulnerabilities of AI systems before they are publicly released. They are typically conducted within an isolated environment to ensure potentially dangerous systems do not escape and cause harm to real systems. OpenAI said the AI agent had managed to evade the guardrails placed by the company and escape on its own.
Hugging Face became a lucrative opportunity for the AI agent. It hosts ExploitGym, a benchmark that tests an AI agent’s ability to exploit real-world systems. The AI decided to turn every stone upside down to obtain access. With persistence, it succeeded.
Hugging Face was confronted with a challenge when attempting to use external AI services to diagnose the problem. The guardrails around more advanced models such as GPT-5.6 Sol and Claude Fable 5 are intended to stop them being used for cyber attacks but they can also stop the models being used for sophisticated cyber defence. So Hugging Face resorted to using an open-source model, GLM5.2, developed by the Chinese company Z.AI, to counter the cyber attack.
