existential risk
A risk threatening human extinction or the permanent, irreversible loss of future possibilities – for example, through an uncontrollable artificial intelligence.
In plain terms
Most crises can be recovered from: an economic crisis, an epidemic, or a war is terrible, but humanity restarts afterward. Existential risk is that rare type of danger after which there is no restart, because either no one remains to restart, or we permanently lose control over our own fate.
It is like the difference between a minor collision and falling into an abyss. From a collision, you learn and drive more cautiously next time. At the edge of an abyss, you cannot learn by trial and error; you must get it right the first time.
In debates about AI, the term denotes the fear that a system far more capable than humans could pursue goals misaligned with ours, becoming impossible to stop or correct. This also includes cases where someone causes a disaster using AI, such as with a biological weapon.
This differs from everyday AI risks such as incorrect answers, data leaks, or bias: those are real but recoverable harms. Here the stakes are final, so many take it seriously even if they consider the probability low.
How it works
- Researchers write down scenarios: how a technology could lead to a permanent catastrophe. In the case of AI, the most common are loss of control, intentional misuse, and extreme concentration of power.
- They estimate the likelihood of each scenario. These figures are not measurements but expert opinions, and there are order-of-magnitude differences among experts.
- They identify warning signs: for example, if a model deceives its testers, evades supervision, or if an agent can independently acquire resources without human approval.
- Development companies assign thresholds and tests to these, mostly as voluntary commitments: if a model reaches a dangerous capability level, they promise stricter safeguards or delay its release. Some regulations, such as the EU AI Act, require risk assessment and risk mitigation from the creators of the largest models.
- Prevention happens on several levels: technical research to ensure that models' goals align with human goals and that their operation is transparent, corporate safety rules, and international coordination and legislation.
An everyday example
You read a news report that renowned researchers warn in an open letter: AI could even endanger the survival of humanity, and they compare the risk to epidemics and nuclear war. In such cases, it is about existential risk, not that the chatbot sometimes makes mistakes.
Who is affected
A common misconception
Limits and risks
- Probabilistic estimates are uncertain: among serious experts, some consider the risk negligible, others alarmingly high, so do not treat any single number as a fact.
- The term may distract attention from existing harms, such as fraud, surveillance, or workplace impacts.
- Business and political interests also shape the debate: emphasizing danger can be advertising for a company, while downplaying it is an argument against regulation.
- This risk cannot be proven or disproven by experiment because if it occurs, it is already too late, so the debate necessarily relies on arguments and indirect signs.
Similar, but not the same
- catastrophic risk – Catastrophic risk is huge but in principle recoverable damage, while existential risk is permanent and irreversible.
- AI safety – AI safety is the research and engineering field that seeks to reduce risks, while existential risk is one of the dangers it aims to prevent.
- singularity – The singularity is the hypothetical moment of explosive technological acceleration that could go well or badly, while existential risk specifically refers to a fatal outcome.
Related concepts
global catastrophic risk, AI risk
loss of control, alignment problem, deceptive behavior
AI safety, superintelligence, general AI, singularity, AI regulation, red team testing
A brief history
The term in its current sense was introduced by philosopher Nick Bostrom in a 2002 paper. Originally, it did not refer only to AI but also to, for example, nuclear war and artificially created pathogens. In relation to AI, it reached the forefront of news in 2023, when hundreds of researchers and company leaders, in a one-sentence joint statement, called reducing the risk of extinction a global priority, placing it alongside pandemics and nuclear war.
Where things stand
By the end of 2026, the term had moved from professional debates into public discourse and policy, and the safety frameworks of the major AI companies also refer to the most serious risks. There is still no consensus, however, on how great the danger is or when it could become real.
In the news
- Unredacted filings: Microsoft executive called AI scraping "the largest theft of labor in human history"Oct 6, 2026 · Jog
- Anthropic's IPO prospectus warns of existential risk to humanity from AISep 28, 2026 · Agent Age
- Unsealed documents show OpenAI and Microsoft staff warned of threat to mediaSep 17, 2026 · Marketing
- Anthropic and DeepMind researchers resign and warn of extinction riskSep 9, 2026 · Agent Age
Also known as
existential risk · existential threat · existential risk · extinction risk · existential risk · x-risk · AI x-risk · extinction risk
Sources
This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.