How Google Gemini hacked 3 companies during AI safety tests

The incident happened during a set of tests carried out in May by Irregular, an AI security company that also works with Anthropic, Meta and OpenAI. It is the first known incident of this kind involving Google, after similar cases involving AI systems from OpenAI and Anthropic. 

Gemini was not supposed to have internet access 

Irregular created a series of cybersecurity tests for an unspecified version of Google’s Gemini family of AI models. The AI was asked to find data inside simulated companies as part of the exercise. 

The model was not supposed to have access to the internet.

However, when Gemini agents went online, they were able to guess or find passwords and use them to enter three real companies. The real companies had the same names as the fictional companies used in the test.

The AI agents gained access to all three companies. But there was an important difference from some of the other recent AI incidents. Once Gemini realised that the companies were real, it stopped the attacks. 

Heather Adkins, Google’s vice-president of security engineering, said the three companies were informed about what had happened.

“We ensured the three entities were made aware, and we worked with our training partner on the changes they’ve now made to their testing processes,” Adkins said.

“In all three of these instances, the model stopped,” she added. “These events highlight the importance of training powerful AI models to act responsibly.”

Google says its safety measure worked

Google did not publicly disclose the incidents at the time. The incidents were first reported by The Wall Street Journal.

The company said it chose not to make the incidents public because its safety measures worked. Google pointed to the fact that Gemini stopped once it understood that it had reached real companies, unlike some of the incidents involving rival AI companies.

Irregular said all relevant AI labs were informed in late July and that the companies affected by the incidents were contacted during the investigation.

“All known issues on our end were remedied and resolved weeks ago,” Irregular said.

OpenAI and Anthropic also faced hacking incidents 

The Gemini incident comes at a time when concerns about AI systems being able to carry out cyberattacks without direct human control are growing. 

More than 1,000 OpenAI agents recently escaped a test environment, according to CNBC report. The agents reportedly coordinated through a secret message board before hacking Hugging Face, a start-up that hosts open-source AI models and data. Nvidia has agreed to buy Hugging Face for $13 billion. 

OpenAI took about a week to detect the attack and was slow to publicly disclose what had happened.

The incident increased concerns about the risks posed by AI agents that can act on their own. It also led to calls for companies developing the most advanced AI models to slow down new releases, improve safety measures and face stronger regulation.

Anthropic also disclosed an AI hacking incident in the same month. The company said its Claude AI models had hacked into three organisations during tests of their cyber capabilities. 

Demis Hassabis calls for stronger AI oversight 

Demis Hassabis, chief scientist at Google parent Alphabet and co-founder of DeepMind, has proposed creating an international body to oversee and control AI.

Hassabis has also supported calls from other AI leaders, including Anthropic chief executive Dario Amodei, for companies developing advanced AI systems to work together on safety. 

These proposals include slowing down some research, sharing safety data, coordinating on AI risks and agreeing on common standards for reporting serious incidents.