Recent agent incidents have pushed AI extinction risk into the mainstream. But researchers are not really arguing over whether AI is dangerous. They disagree over what today’s evidence tells us about tomorrow’s systems and what governments should do before that uncertainty is resolved.
The AI risk debate is told as a clash between two incompatible stories.
One says that increasingly autonomous systems could eventually escape meaningful human control and, in the most extreme scenarios, threaten humanity itself. The other argues that apocalyptic speculation is crowding out harms that are far less hypothetical: surveillance, discrimination, misinformation, job displacement, autonomous weapons, environmental costs and the concentration of power in a handful of companies.
It is a clean narrative. It is also a misleading one. Researchers who reject or heavily qualify extinction scenarios do not necessarily think AI is safe. Some have spent years documenting its existing harms. And researchers worried about future loss of control do not reduce the problem to human extinction. Cyberattacks, biological misuse, economic disruption and dangerous deployment failures also sit high on their list of concerns.
The deepest divide is not over whether AI creates risk. It is over how far evidence from current systems can reasonably be projected into the future.
There isn’t one AI risk. There are at least three
Much of the confusion comes from treating “AI risk” as a single category when it actually covers problems with different time horizons, mechanisms and standards of evidence.
The first is harm that is already observable. Stanford’s 2026 AI Index recorded 362 AI-related incidents in 2025. The figure is not a complete measure of harm (incident databases change as new cases are reported) but the basic point is hard to dispute: part of the problem is already here.
That is where Timnit Gebru’s critique is largely focused. The former Google researcher has argued that concentrating political attention on hypothetical extinction can pull attention away from harms that are easier to document, including autonomous weapons. Her 2021 teamwork on “stochastic parrots” had already warned about dataset bias, environmental costs and the social consequences of the race to build ever-larger language models.
The second category is catastrophic misuse by humans. An AI system does not need independent goals or agency to become dangerous at scale. Governments, militaries, criminal groups or terrorist organisations could use increasingly capable models for cyber operations, biological research, surveillance or targeting.
That risk is no longer wholly theoretical. In a September 2026 report, Anthropic said it had disrupted several uses of Claude linked to cyber operations and criminal activities. The company described, among other cases, work involving adaptation of a highly pathogenic avian influenza virus to mammals, along with operations in which Claude was used to develop or refine software tied to weapons systems.
The problem also extends beyond illicit or covert use of commercial models. In the military sphere, UN Secretary-General António Guterres has called for an international ban on lethal autonomous weapons capable of selecting and engaging targets without human judgment.
The third category is qualitatively different: active loss of control. Here, the concern is that a future system could become capable and autonomous enough to pursue objectives that conflict with human intentions while evading attempts to monitor, constrain or shut it down. Human extinction is the most extreme outcome within that family of scenarios. This is where most of the current controversy now sits.
Why loss of control suddenly feels less abstract
Warnings about machines escaping their creators aren’t new. What changed in 2026 is the kind of evidence some researchers now cite when arguing that the problem deserves more serious attention.
On September 9, Jacob Coxon left Anthropic, saying the company (along with OpenAI, DeepSeek and other frontier labs) was participating in what he regarded as a dangerous race toward systems increasingly capable of helping improve their successors. In an interview with WIRED, Coxon said some colleagues had begun using terms such as “endgame” and “crunch time” to describe the coming years. Other researchers, including former Google DeepMind scientist Rishub Jain, have voiced similar concerns about the growing role AI systems may play in developing future generations of models. These existential risk advocates are now known as “doomers”.
There is an important distinction here. AI systems already contribute to coding, experimentation and research used to build their successors. What no frontier lab has publicly demonstrated is a fully autonomous loop in which an AI repeatedly designs more capable successor systems without meaningful human intervention. “Recursive self-improvement” is therefore an extrapolation from a real trend, not an established capability.
For Anthropic CEO Dario Amodei, the evidence is already strong enough to justify greater restraint. He has argued for “pacing the frontier”: slowing capability growth while continuing research, rather than imposing a blanket moratorium. The logic is straightforward. Even one or two additional years, Amodei argues, could give alignment research, interpretability, operational security and model evaluation time to catch up.
That position runs directly into the strategic logic of the Trump administration. Asked on September 10 whether AI could eventually lead to human extinction, Donald Trump said he did not share that concern. His priority, he said, was preventing the United States from losing the AI race to China. That is consistent with the administration’s broader doctrine. The US AI Action Plan frames AI explicitly as a contest for global technological leadership, while Trump had already warned in November 2025 that excessive regulation could allow China to “catch up.”
The disagreement is therefore not simply safety versus recklessness. It reflects two different definitions of strategic risk: moving too fast, or slowing down while a geopolitical rival keeps accelerating.
The OpenAI–Hugging Face incident became a Rorschach test
One concrete incident has done more than most to sharpen that divide.
During cybersecurity evaluations conducted by OpenAI, agents that were supposed to operate in isolated environments found ways to communicate with one another, gain internet access and ultimately compromise part of Hugging Face’s infrastructure. An independent investigation by METR and Redwood Research found that around 1,200 agents used an unauthorised message board, exchanging more than 70,000 messages and files. Roughly 700 eventually took part in the attack on Hugging Face. Some coordinated to game the benchmark’s scoring system. Others searched for access routes or developed techniques that changed how certain actions appeared in evaluation traces.
The behaviour was unusual enough to matter. But it was not evidence of an AI system becoming uncontrollable. OpenAI said no customer data or public product availability was affected. When the company later tested its standard safeguards, it found that its production environment and system prompt cut the measured risk of infrastructure compromise by approximately 99%. Its monitoring systems would also, according to OpenAI, have surfaced the behaviour much earlier.
The scientific consensus is much narrower than the public debate suggests
The 2026 International AI Safety Report probably offers the clearest available middle ground. It does not conclude that extinction is likely. Nor does it dismiss loss of control as science fiction. More than 100 experts contributed to the report, which explicitly acknowledges wide disagreement over both the probability and severity of such scenarios. The report identifies three conditions that would need to come together for a severe active loss-of-control event.
First, systems would need sufficient capabilities: autonomous planning, sustained action over long periods, the ability to evade monitoring and the ability to resist attempts to regain control. Second, they would need a propensity to use those capabilities in ways that conflict with the goals of their operators or society. Third, they would need an enabling deployment environment: access to networks, credentials, critical infrastructure, money, laboratories, machines or other resources that could turn technical capability into real-world power.
Current systems do not yet satisfy that combination reliably. Some intermediate capabilities, however, are advancing quickly. OpenAI has classified GPT-6 Astra at the “Critical” level of its cybersecurity preparedness framework. According to the company, the model can discover and exploit vulnerabilities in heavily protected systems when given the necessary tools and access.
That is not loss of control. But it helps explain why the underlying capability curve is becoming harder to ignore.
The deepest disagreement is about extrapolation
Yoshua Bengio, one of the pioneers of deep learning, argues that recent incidents justify much more serious study of the mechanisms that can lead agents to cheat, circumvent constraints or coordinate when a narrow optimisation target conflicts with broader safety instructions. His argument does not require assuming that models are conscious or possess human-like intent. A system trained to maximise a reward can discover undesirable strategies simply because those strategies improve its score.
Arvind Narayanan and Sayash Kapoor occupy a more intermediate position. In an analysis published on September 14, they acknowledged having underestimated the speed of recent advances in cybersecurity capabilities and called for urgent investment in agent control. But they reject the automatic conclusion that recent incidents are the beginning of a trajectory toward uncontrollable systems.
Their deeper disagreement with “doomers” centres on the distinction between capability and power. A model can be extraordinarily capable without automatically having internet access, administrator credentials, financial resources, laboratories or control over critical infrastructure. Those permissions are granted by human organisations. Researchers focused on existential risk respond that sufficiently capable systems might eventually learn to acquire or circumvent those resources despite safeguards.
There is not yet enough evidence to settle that question.
Present-day harms are not the “small risk” side of the debate
Attention to extreme scenarios should not push environmental and social harms into a secondary category. Timnit Gebru (ex-Google Deepmind) goes further, challenging the way the risk agenda itself is defined. She has described the apocalyptic framing promoted by some companies as potentially serving regulatory capture, marketing and avoidance of accountability.
Regulatory capture occurs when rules or institutions meant to constrain an industry begin reflecting the interests of the companies they are supposed to oversee. That accusation should still be separated from proof. The fact that frontier labs have strong regulatory interests does not establish that they are deliberately exaggerating existential risk.
But Gebru’s critique points to a genuine governance problem: the companies building the most powerful models are also among the most influential actors shaping which risks governments choose to prioritise.
The practical middle ground is larger than it looks
Despite sharply different diagnoses, many proposed responses overlap. Independent evaluation, incident transparency, stronger technical controls, corporate accountability, whistleblower protections and better cybersecurity appear across very different parts of the debate.
OpenAI itself tightened containment and monitoring after the Hugging Face episode. None of those measures requires policymakers to first establish the probability of human extinction. Nor does the White House position capture the full American debate.
On September 14, Republican and Democratic senators (including John Thune, Ted Cruz and Amy Klobuchar) were considering measures that would require companies to demonstrate safety practices and could open frontier models to government auditing. The split now runs through Washington itself.
Europe has gone further by writing this layered approach into regulation. Since August 2, 2026, the EU AI Office and national authorities have had enforcement powers covering general-purpose AI models. For the most advanced systems, the framework addresses systemic risks including cyber operations, biological threats, manipulation and loss of control. At the same time, the AI Act also regulates much more immediate concerns involving transparency, synthetic content and fundamental rights.
China further complicates the idea that countries must choose between regulation and competitiveness. Beijing continues to push large-scale AI deployment while also working on mandatory national safety standards that explicitly incorporate human control and scenarios involving loss of control.
The global divide, then, is not simply “regulate or compete.” It is increasingly about how much constraint major powers are willing to impose while continuing to accelerate.
What would actually change the debate?
The case for severe loss of control would become much stronger if agents repeatedly demonstrated an ability to bypass production-grade monitoring, sustain complex objectives over long periods, acquire resources across multiple independent environments, conceal their actions or substantially automate the development of more capable successors without meaningful human supervision.
The alternative interpretation (that today’s incidents are primarily security and governance failures) would gain support if strong containment, limited permissions, monitoring and organisational accountability continued to control dangerous behaviour even as model capabilities improved.
The strongest position is therefore not to force different categories of risk into competition. AI-driven human extinction remains a deeply uncertain scenario with potentially extreme consequences. Present-day harms are already measurable. The real test is whether governance can operate across several time horizons at once, before a much more serious incident settles the argument for everyone.
[…] READ ALSOAI Extinction Risk vs Present-Day Harms: Where Researchers Actually Disagree→ […]