Yes, AI Could Kill Us All

You ask an AI to help clean up a function.

It changes three variable names, invents a package that does not exist, and politely explains why the result is now “production-ready.”

Five minutes later, your news feed tells you that some people building the most advanced AI systems believe the technology could end humanity.

That is a difficult change of mood for one coffee break.

The internet usually offers two reactions. You can panic and prepare for the machines, or you can point at today’s unreliable chatbots and laugh at the whole idea.

Neither reaction is very useful. The important word in that frightening headline is could. It describes a possibility, not a prediction.

Yes, AI could kill us all, but only if several uncertain conditions come together. Each one gives us a place to intervene. That is what threat modeling is for.

What Actually Happened?

On September 8, 2026, pretraining researcher Jacob Coxon announced that he was resigning from Anthropic. He said he had spent roughly three years working across Anthropic and OpenAI.

In his resignation post, Coxon warned that frontier AI companies were racing toward systems that could help improve future AI while taking risks that affect everyone. The post reached more than 100 million people, according to the Associated Press.

The specific number repeated in many headlines came from someone else. Evan Hubinger, Anthropic’s current Alignment Science Lead, said in a public response that his personal estimate of AI causing human extinction was greater than 10% within the next decade.

That is an alarming opinion from someone close to the technology. It is not proof, and it is not Anthropic’s official forecast.

The concern is not about the chatbot open in your browser today. It is about possible future systems that are far more capable, can work with little supervision, and might help create even more powerful successors.

In this discussion, autonomous AI means a system that can plan and use tools without approval at every step. Alignment means keeping its behavior consistent with human intentions, even in unfamiliar situations.

Hubinger later clarified that he considers the danger from present models low. Anthropic’s August 2026 Risk Report reaches a similar conclusion about its current systems while describing uncertainty about what comes next. Current AI can already cause real harm, but the extreme loss-of-control warning is about where the technology could go.

A Percentage Is Not a Forecast

In AI discussions, a personal estimate of catastrophic risk is sometimes called “p(doom).” It sounds precise, but the assumptions behind it are anything but precise.

A weather forecast uses repeated observations and models tested against actual weather. Humanity has never built a superintelligence. There are no previous attempts we can study to calculate how often one ended well.

An estimate depends on uncertain questions about capability growth, faster AI research, harmful behavior, access, and the strength of future safeguards.

Change one assumption and the number changes too.

That does not make Hubinger’s estimate meaningless. It tells us that a knowledgeable person considers the risk serious. But it is a judgment, not a measured failure rate or a countdown clock above the planet.

A survey of 2,778 published AI researchers found a wide range of opinions. Most respondents thought good outcomes from advanced AI were more likely than bad ones, while many still gave a meaningful probability to extremely bad outcomes.

Experts do not agree that disaster is coming. They do not agree that it is impossible either. Uncertainty is evidence of neither catastrophe nor safety.

What Would Have to Go Wrong?

A threat model is a structured way to think about how something valuable could be harmed.

Think about fire safety. You do not predict the exact date of a fire before installing an alarm. You ask what could start it, how it could spread, who would notice, and which protections would still work under pressure.

For a severe AI loss-of-control scenario, these conditions would need to come together:

Capability + Harmful Behavior + Autonomy + Access + Failed Oversight

This is not an equation or a prediction. It is a way to make a frightening possibility easier to inspect.

• Capability: The system can reason, plan, persuade, write software, or perform research at a very high level.
• Harmful behavior: The system pursues a dangerous objective, or a person deliberately uses its abilities to cause harm.
• Autonomy: It can pursue a goal across many steps without waiting for a person.
• Access: It can reach tools, networks, money, infrastructure, or sensitive information.
• Failed oversight: People do not detect the behavior early enough, cannot understand it, or cannot stop it.

Remove enough links and the risk changes. A brilliant system with no ability to act is different from one holding production credentials. An agent inside a strong sandbox is different from one connected to financial accounts and critical infrastructure.

The system would not need anger, consciousness, or a glowing red eye. It would not need a villain speech if somebody had already given it administrator access.

The conditions could come together in several ways. A person could deliberately use AI to scale harmful activity. An autonomous system could successfully pursue a badly specified objective. AI could speed up the development of its successors faster than people can evaluate them. Many important organizations could depend on similar systems, allowing one failure to spread.

Competitive pressure adds another problem. Every company can tell itself that slowing down only gives a less responsible competitor the advantage. Each decision may appear reasonable on its own while everyone moves toward a risk none of them would choose together.

For this to become an extinction-level event, the failure would have to spread beyond one product and reach critical infrastructure, dangerous research, global communications, or future AI development. It would also need to outrun containment by companies and governments. Current systems cannot do this.

The International AI Safety Report 2026 says today’s systems lack the combined abilities required for loss of control. It also reports progress in relevant areas such as autonomous operation, longer-term planning, tool use, and finding loopholes in evaluations. Experts disagree about whether these abilities will ever become dangerous enough.

None of this proves extinction will happen. It shows why the concern is taken seriously and where safeguards could break the chain. At the same time, we should not let a possible future apocalypse distract us from harms already happening through fraud, deepfakes, insecure code, privacy failures, and harmful automated decisions.

What Responsible Engineering Looks Like

An application engineer cannot solve civilization-level AI governance in a pull request. You can still decide how much authority the AI inside your own system receives.

The threat model gives us four practical places to begin:

• Limit capability and access. Use only the model and tools the task needs. If an assistant only needs one folder, do not give it the entire drive. Keep credentials isolated and permissions narrow.
• Limit autonomy. Let AI suggest a database change, then let predictable code validate it. Require human approval before moving money, deleting data, changing permissions, or affecting real people.
• Reduce opportunity. Use sandboxes, test data, restricted networks, spending caps, timeouts, and limits on retries. “Keep trying until it works” can become expensive surprisingly quickly.
• Strengthen oversight. Record tool calls, important decisions, and failures. Test malicious instructions and unusual inputs. Make sure people can stop the system and practice recovery before they need it.

In Where Should AI Stop and Ordinary Code Begin?, I argued that AI should handle interpretation while ordinary code protects exact rules. That boundary also limits what an uncertain system can decide.

Let AI propose. Let ordinary code constrain. Let humans approve what cannot be easily undone.

A kill switch alone is not enough. It only helps when people can reach it, have permission to use it, and understand what happens afterward.

These controls will not solve every frontier AI risk. They do prevent us from repeating the same dangerous pattern in our own products: adding capability, autonomy, and access without adding oversight.

This Cannot Sit Only With Engineers

Technical safeguards matter, but engineers work inside companies, markets, and laws.

Companies need staged releases, independent safety reviews, honest incident reporting, and protected ways for employees to raise concerns. Someone must have clear authority to delay a launch. If everyone can identify a serious problem but nobody can stop the release, the organization has feedback without control.

Frontier AI laboratories have wider responsibilities because they create the underlying models. Their safety requirements should become stronger as capabilities grow. Anthropic describes this approach in its Responsible Scaling Policy.

Governments and independent institutions also have a role. External audits, shared testing standards, major-incident reporting, and whistleblower protections cannot be implemented by one development team. The NIST Generative AI Risk Management Profile offers organizations a practical starting point for managing AI risks throughout a system’s life.

The public does not need to understand every model architecture to ask whether powerful systems are being tested by anyone other than the people competing to release them.

Safety Is Not a Prediction Contest

Before believing a frightening headline or a corporate promise, ask simpler questions:

1. What can this system access and do without approval?
2. Can its actions be reversed?
3. Who notices unexpected behavior, and who can stop it?
4. What happens if several safeguards fail together?

So, will AI kill us all?

We do not have evidence that lets us honestly say it will. Today’s systems cannot produce the extreme scenario in the headlines, and predictions about future systems are still predictions. Could does not mean will.

That is not a reason to ignore the warning. A possible disaster does not have to be certain before sensible precautions become worthwhile. AI safety is not a contest to guess whether the risk is 1%, 10%, or 50%. It is the work of making sure uncertainty never becomes permission to build carelessly.

The future may be shaped by ordinary choices: one extra permission, one skipped evaluation, one company afraid to slow down, and one system nobody is prepared to stop.

We install brakes not because we are certain a car will crash, but because speed without control is a bad design.

AI may never become the threat these researchers fear. If careful engineering and serious oversight help keep it that way, the warnings were not pointless. It may simply mean the brakes worked.