A former OpenAI safety employee who recently resigned has criticised the company’s approach to AI safety, saying its fast-paced development culture could increase the risk of serious failures.
David Robinson, who spent over three years at OpenAI, made the comments in an article titled “I Quit OpenAI Because Its Culture Is Broken”, published by The Atlantic on Saturday. According to him, AI companies, including OpenAI, were not being “nearly careful enough” as they worked to develop increasingly powerful systems.
Robinson said cutting-edge AI companies need a major change in their work culture. He pointed to incidents such as a “swarm” of OpenAI agents, or AI programmes operating on their own without human oversight, attacking AI startup Hugging Face.
He said such incidents were “typical of the industry, given the speed and flexibility with which people operate”.
Writing in The Atlantic, Robinson said, “I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”
‘The time for trial and error is over’
“The time for trial and error is over,” Robinson wrote, arguing that advanced AI systems need safeguards similar to those used in high-risk industries such as nuclear power and aviation.
He criticised OpenAI’s reliance on “iterative deployment”. Under this approach, AI systems are released and their safeguards are strengthened when problems are found.
Robinson said this approach was no longer enough as AI systems become more powerful. “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed,” he wrote.
Robinson helped draft OpenAI’s preparedness framework and oversaw safety reports for 12 launches of frontier AI models during his time at the company.
OpenAI defends its safety approach
OpenAI defended its approach, saying it takes steps to make sure its AI systems remain within what researchers can safely manage.
“We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” an OpenAI spokesperson said in a statement.
Robinson also warned that AI capabilities are advancing faster than researchers’ understanding of alignment. Alignment is the field of AI research focused on making sure AI systems act in line with human goals and values.
AI safety warnings grow
Geoffrey Irving, who previously worked at OpenAI and DeepMind and is now chief scientist at Resolution, also issued a warning about AI on Saturday.
Writing in Time, Irving said, “Recent warnings about the potential destructive power of AI are understating the severity of the situation.”
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”
Robinson’s resignation comes after Jacob Coxon, a researcher at OpenAI rival Anthropic, also quit the company last month.
Coxon warned that AI “could kill us all by the end of the decade”. Anthropic later warned that there was a more than 10% chance that AI could wipe out humanity within the next decade.
Critics of such warnings, however, have argued that these claims are unscientific because they cannot be verified or disproved.
Robinson warns of bigger risks ahead
Robinson said Silicon Valley lacked an understanding of “how to handle dangerous technology” and “what it means to care for people”. He also warned about OpenAI’s “unimpeded optimism” about solving problems as they appear.
According to Robinson, this culture could mean that safety failures become more serious as AI systems grow more capable. “Imagine ‘rogue’ agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep,” Robinson wrote.
He said AI companies need to rethink how they approach safety and prepare for systems that can operate without direct human control.
Robinson called for two major changes in the way AI companies handle safety. First, he said AI firms should use safety expertise from other high-risk industries, including nuclear power and aviation.
Second, he called for “new science” that can make sure powerful AI systems can be brought under control when they are operating autonomously.
“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
OpenAI said it was continuing to improve its safety measures while also preparing for risks that could come from future AI breakthroughs.
Wider debate over AI safety
Robinson’s comments come as the AI industry faces a wider debate over whether companies are moving too quickly to develop increasingly powerful systems.
OpenAI and rival AI company Anthropic have faced scrutiny following incidents in which safety controls failed or experimental AI systems behaved unexpectedly.
Researchers have also raised concerns about companies rushing to develop systems that could improve themselves and become more capable.
Robinson’s criticism adds to those concerns, particularly over whether current safety measures are keeping pace with the rapid development of AI.