The biggest talking point these days, dominating conversations about AI, concerns existential threats to humanity. Since July, there’s been a shrill escalation in warnings. Diverse views coalesce around how powerful AI can become, whether humans are still in control, and how fast we should move.
With good reason. On July 21, OpenAI models breached controls and compromised Hugging Face’s production systems. On July 30, Anthropic revealed that that there were three incidents when Claude models accessed systems of outside organisations via the internet. Meta disclosed a similar breach on August 5. And Google confirmed three cases of corporate systems being affected on September 18.
Comments made by AI industry leaders are revealing.
Anthropic CEO Dario Amodei, in his essay “We must pace the frontier”, points to the misuse of AI for cyberattacks, bioterrorism, and economic disruption, despite its obvious benefits. And the danger of agent swarms commandeering the internet without dependable safeguards, because this technology is accelerating faster than our ability to keep it safe.
Nobel laureate Geoffrey Hinton, often called the godfather of AI, told BBC on September 9 that as humanity has never created entities smarter than humans, the risks are difficult to predict. When asked if there was a greater than 10% chance of AI killing humans, he seemed to think that isn’t an unreasonable estimate.
Demis Hassabis, CEO of Google DeepMind and Isomorphic Labs, also a Nobel laureate, published a note in July, called “A framework for frontier AI and the dawning of a new age”, in which he says that artificial general intelligence (AGI), a system with all the cognitive capabilities of the human brain, is just a few years away. He says we have found ways to make sand think, with the magnitude of impact being 10x the Industrial Revolution at 10x the speed. He calls for a US-led global watchdog to put governance mechanisms in place, commensurate with AGI’s unprecedented power.
Sam Altman, CEO of OpenAI, in a companywide meeting, told his employees that OpenAI could potentially pace its AI development in conjunction with other AI labs. The company’s top scientist, Jakub Pachocki, recently wrote a post, warning about dangers from AI, seeking voluntary slowdowns until shared safety bars are established.
Anthropic researcher Evan Hubinger set the cat among the pigeons when he said, “We really do believe AI could kill all humans.” Jacob Coxon, Anthropic researcher, quit the company on September 8, citing his fears in posts stating that AI labs are gambling with humanity’s future. Other AI safety researchers followed suit. Josh Engels, part of Google DeepMind’s AGI safety team, quit to join independent evaluator METR. He warns that recursive self-improvement through which AI systems build increasingly capable successors could become dangerous, if alignment does not keep pace with gains.
Elon Musk has been warning the world about existential risks for years now, likening it to a nuclear situation. While continuing to pursue AI as he thinks it can become the greatest source of global abundance, he reiterated in an interview with The Economist in July that it will take about 10 years till humans lose control.
On the other side of the fence, Nvidia Chief Jensen Huang rejected predictions of an AI apocalypse as “complete nonsense”. He believes AI outages are an engineering problem, which can be safely managed, and there’s “zero chance AI will end the world by 2030”.
Mark Zuckerberg of Meta published a lengthy manifesto in August, arguing that concerns over AI are overstated and that the bigger danger is the concentration of control in a single entity or government.
President Donald Trump has been unequivocal about his stance, dismissing fears over rogue AI as a hoax, and claiming to be the hoax buster. He says that limiting AI technology is part of a sick conspiracy, echoing his well-known views on climate change.
Motivations to support or negate this existential debate about machine intelligence surpassing human intelligence throw up compelling aspects.
Beijing reads this as an attempt by the US to institutionalise the lead it has at this time, while President Trump isn’t willing to slow AI in the US because China will get ahead.
Venture capitalists are also calling out that market leaders advocating a slowdown want to preserve their lead.
Huang could well be overwhelmed by the commercial implications of Nvidia supplying much of the computing infrastructure. Meta’s broader strategy of supporting open AI models could be influencing Zuckerberg’s position.
The biggest paradox, however, is that the people most worried about AI destroying humanity, are the people building it.
Meanwhile, a lawsuit filed on September 18 in California’s Northern District Court argues that leading AI companies violated anti-trust laws when they agreed to slowdown efforts, as it would reduce the value consumers would get for paid subscriptions.
The risks are many. Including the unknown unknown of transformative technologies that can process, reason, and predict, without an ethical core, empathy, or any accountability. Plus, the enormous intelligence divide between countries, institutions, and individuals who have access to AI models, computing power, and data, and those who don’t. Add the cognitive atrophy that creeps in when we get comfortable relying too much on machines rather than our own brains. And the consequences when even one element in the AI value chain spanning chips, energy, data, models, compute, capital and more, becomes scarce.
The dominant themes remain. The accelerators are convinced they are on the threshold of abundance while naysayers feel the extinction of the human race is at hand. Perhaps it calls for the adoption of healthy scepticism tempered by robust ambition. The jury is out.
Disclaimer: The views expressed are the author’s own and do not reflect the official policy or position of Financial Express.