The week “AI existential risk” went mainstream

What does it mean to “win the AI arms race” when humanity is at stake?

human ingenuity juxtaposed with nature’s beauty… from a trip to the Bay this week

The Artificial Intelligence (AI) discourse is moving so fast with so many moving parts that it’s incredibly difficult to separate signal from noise. Last week something significant happened: the mostly-AI-insider conversation about what is called “existential risk” — namely, the possibility that advanced AI could irreparably destroy humanity — burst into mainstream conversation. Specifically, an employee at Anthropic — one of the main AI labs and the maker of Claude — announced his resignation with this declaration:

I want to do a short post today attempting three things (writing from an outsider's perspective trying to understand the AI landscape):

1. A quick explainer with how I understand what is going on: what is AI existential risk and how freaked out should we be?

2. The bridge to individual agency: why I think AI existential risk reveals something fundamental about our economic system.

3. What this means for how you can be part of solutions.

TL;DR: The bad news: the risk is real; we are right to feel worried. But it requires several specific things to go wrong together, and we’re not there yet. This isn’t preordained.

The mixed news: while the technology is new, AI existential risk isn't actually a new kind of problem. Rather, I believe it reveals the same underlying systemic failure that is driving many of the other crises we face: the deep illogic baked into our current system design that values the pursuit of profit over the preservation of life.

Which means there is good news. We have a unique transpartisan opportunity to come together to make different choices: both at the level of commonsense AI governance, and in the role each of us has to play in creating a culture that is aligned with life.

---

AI existential risk goes mainstream: what just happened?

First, some context: five major U.S.-based AI labs drive most of the conversation. These companies build and train what are called large language models (LLMs) that they make with their own proprietary software and algorithms; they are competing with each other for market share. The main ones worth knowing for today’s post:

• OpenAI (Sam Altman) — ChatGPT

• Anthropic (Dario Amodei) — Claude

• Google DeepMind (Demis Hassabis) — Gemini

• Meta AI (Mark Zuckerberg) — Llama

• xAI (Elon Musk) — Grok

• DeepSeek (China, state-supported), name of both company and model

Here’s a brief synopsis of the dots I’m connecting that pushed AI existential risk into the mainstream conversation, and the role of some of these key players:

• Sept 3: NY Times publishes a cover story with alarming new details about the July Hugging Face attack, when rogue AI from OpenAI breached the systems of another AI company called Hugging Face

• Sept 9: Jacob Coxon posts his resignation from Anthropic on X, warning that both major labs are “racing straight to self-improving superintelligence and gambling with our lives” — the post reaches 100M+ views and is echoed by other insiders

• Sept 12 — Dario Amodei (Anthropic CEO) publishes “We Must Pace the Frontier,” calling for the industry to deliberately slow capability gains and committing Anthropic to continuous third-party evaluation

• Sept 13 — Sam Altman and Elon Musk both publicly endorse Amodei’s call to pace the frontier within about a day of each other

• Sept 15 — Bernie Sanders (left wing populist) and Steve Bannon (far-right populist) share a stage at the “Pro-Human Assembly” in DC, with Sanders announcing superintelligence-ban legislation and calling for a Trump-Xi AI treaty

This is all happening prior to a high-level meeting next week between President Trump and Chinese President Xi, where AI governance is on the agenda.

Understanding AI existential risk: what are we afraid of?

Deep breath: this gets grim. First: take solace in knowing that none of this is preordained, and it is all avoidable. While there is risk, that does not mean this is going to happen. Second: this information is definitely activating to the nervous system: please take care of yourself as you proceed. I recommend taking a break as needed to be outside, to touch a tree, to ground yourself. Let’s proceed together.

Existential risk is a term of art in the field of global risk analysis, defined by Nick Bostrom in a 2002 paper:

One where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.

I took a class on existential risk in grad school; the paradigmatic example at the time was an asteroid strike.

The existential risk of AI as I understand it follows a specific logic flow. What’s important to understand is that these are all separate but related: it’s not necessarily sequential or if-then. The existential risk that we are most worried about requires all five steps:

1. Capability: Can it?

   Does the AI have sufficient capabilities to take consequential action at global scale? [Note: AI insiders tend to associate this with achieving what is called “artificial general intelligence” (AGI) or “superintelligence” (ASI). But while AGI and ASI describe increasingly general levels of capability, neither is itself the threshold for existential risk.]

2. Motivation / alignment: Will it pursue its own objectives?

   Would the AI pursue goals contrary to human interests? What’s important here is that it doesn’t require malicious intent: the famous “paperclip” thought experiment demonstrates this risk (if the objective is “maximize number of paperclips,” then the AI could decide that humans are an unnecessary obstacle to the goal of more paperclips).

3. Autonomy: Can it act on its own?

   Can the AI independently pursue that objective? Can it initiate action without human prompting, and can it direct and control other AI systems in pursuit of its objectives?

4. Loss of human control: Can we stop it?

   Can humans still reliably prevent or redirect the AI? This requires that we are aware of the misalignment (#2 above), aware of the autonomous actions (#3), AND able to intervene to prevent the AI from pursuing its objectives. If not…

5. Impact: Can it destroy us?

   If the AI acts, can the consequences become catastrophic to the point of existential threat for humankind? The main mechanisms imagined are physical (capturing and unleashing weapons of mass destruction like global nuclear war); biological (developing and disseminating lethal pathogens at global scale); and digital/infrastructural (disabling power grids, water supplies, transportation and food systems).

So: yeah, that’s terrifying. And it’s terrifying because it’s plausible. While the Hugging Face attack does NOT demonstrate an existential threat, is does show that some of the elements we've historically treated as hypothetical—AI systems acting autonomously, pursuing objectives humans didn't explicitly authorize, and operating in ways that can evade ordinary oversight—are already happening.

Best guesses range wildly at the probability of this existential risk, but the disturbing thing that Jacob was pointing to in arguing that lab leaders are “gambling with our lives” is that pretty much all experts agree that the risk is above zero—with some estimates ranging alarmingly high. In a sign of how deeply lost Silicon Valley culture is, there is actually a term of art/mathematical formula for this, expressed as: p(doom). Namely, the probability of existential destruction. And there are robust debates over how high the probability is, with different weights implying different strategic choices.

I can’t believe I even have to say this, but here we are: if the possible consequence is the destruction of humanity — the acceptable level of risk should be zero.

Another deep breath. Reminder: we are capable of developing safeguards at every level. We can choose a path that eliminates this risk.

Exposing the rot: AI existential risk is the symptom pointing to the problem

One might reasonably wonder why an AI lab lead — if he genuinely believes that there is a non-zero existential risk to humanity — doesn’t unilaterally stop building the capacity of advanced AI models.

The argument from Silicon Valley often invokes the “Moloch problem” or the language of “multipolar traps” as an explanation: basically the idea that each individual/organization is incentivized to pursue outcomes that leave everyone worse off. While one lab lead may want to slow down for moral reasons, his financial incentives and the threat of competition from other labs push him to speed up. In game theory people intuitively apply the prisoner’s dilemma as the default mental model: dominant strategy is full speed ahead.

What all of these discussions gloss over is that we are comparing life to money. The cost of unilaterally choosing to pause is measured in loss of market competitiveness: in other words, loss of money. At the geopolitical level in the context of U.S.-China rivalry, that same logic just wears a different uniform — money becomes a proxy for military and strategic advantage, and “falling behind” is viewed as a national security threat. But the underlying comparison is the same. The cost of continuing to speed — even if everyone else stops! — is the possible destruction of life on Earth. The only possible payoff of speeding ahead in anything other than the short-term is if you can somehow reliably prevent that outcome from happening… which is only possible if you don’t speed ahead. This is why the language of an “AI arms race” is misleading: there is no finish line, and no one “wins.”

These are the stakes: the possibility of short-term financial gains vs the possibility of destroying human life on earth.

In other words: AI existential risk is an extreme expression of the same underlying pattern that produces our other crises—our systems repeatedly reward the pursuit of economic value even when doing so undermines the conditions for life.

As Pope Leo warned in his recent encyclical:

Technology is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it.

This is not surprising: we have made the very mistake we are worried advanced AI will make. We have misaligned objectives, and confused means with ends.

Money, markets, technology, productivity, and economic growth were supposed to be means to human and ecological flourishing. Instead we have become expert at increasing money and productivity… at direct cost to human and ecological survival. This is the fundamental design flaw that has brought us to this moment of intersecting crises: our systems systematically give greater weight to economic value than to the conditions that sustain life. We should not be surprised that our most advanced technologies are repeating our mistake.

Stepping back from the cliff-edge

We can make different choices. Within the realm of AI, we as a species can choose a path which refuses to accept existential risk to life on Earth. Here’s what that might look like:

1. An immediate pause on all further AI development to allow enough time for guardrails to be put in place.

2. A democratic decision-making process that represents the voices of all those affected, tasked with developing a set of guardrails to guide AI governance going forward.

3. An effective enforcement mechanism to ensure compliance with life-protecting protocols.

The good news is these steps are straightforward, and there is growing momentum in favor of all of them. The bad news is that mustering the political will to execute these simple tasks is a monumental challenge, and we are a long way from where we need to be.

The events of the last week have at long last galvanized real energy, at least rhetorically. Prominent global leaders have explicitly called for a transnational regulatory body modeled on those created to address other collective action problems (e.g. nuclear non-proliferation). There is growing consensus on the need for a pause (and even AI lab leaders like Dario, Sam, and Elon agree on the need at minimum for slowing down frontier model capacity development). And there is widespread agreement that the fate of humanity should not be left in the hands of five men in Silicon Valley.

Here I would go further than those arguing for a “pro-human” approach to AI: I fear it is an overly anthropocentric worldview that has gotten us into this mess. Rather I want us to take a more capacious view that honors all life on Earth, and find ways to explicitly represent the equities of the more-than-human world in our decision-making (I’m intrigued by Phoebe Tickell’s call for an Interspecies Council as one innovative approach to AI governance).

Looking in the mirror: acting from agency

Most of us don’t feel a lot of agency when it comes to the problem of existential AI risk: it feels too elite, too technocratic, too tied up in global power politics. Even if we had the access it’s hard to imagine successfully influencing Donald Trump to do the right thing in the Xi meeting next week. It’s admittedly a huge lift to create a new transnational entity with the capacity to enforce binding constraints on state actors and private companies.

But if we accept that this is merely the most immediate expression of a core flaw at the foundation of all of our systems, then we actually have tremendous agency. This is where I want to close today: with an invitation to take action wherever you touch the system.

First, an important note on justice. Responsibility and accountability is NOT evenly distributed. Those at the top with the most access to structural power in these systems have a massive debt to life which they must do their best to repay. And we cannot wait for them to do the right thing: by definition those most entrenched in these systems are the most blinded by them. Part of our work is participating in movements that insist on transformation: inviting right action without waiting for it.

But we all — at least those reading this newsletter — participate in these systems. We have an opportunity to choose life over profit, in as many places as we can. As shareholders, as board members, as voters, as AI users, as friends: we can do the best we can to help each other align our actions with life.

None of us want this outcome — not even those driving us toward the cliff edge. They are, in a real sense, acting logically: their choices make sense inside a culture that currently treats this level of risk as an acceptable — even necessary — cost of doing business. But culture isn't fixed — it's something we make and remake together, in every choice about what we'll tolerate and what we won't.

Which means we have real power here, even if it doesn't feel like it: as we build a culture in which this tradeoff is unconscionable rather than unfortunate, we change what “acting logically” requires of the people at the wheel. We make it easier for them to act in accordance with their consciences, not their quarterly returns. So while we need to take action to directly influence their decisions (those involved in the AI ecosystem), we also need to take action to change the culture that enables those decisions. We all have a role to play.

---

I want to give the last word here to the late great Wendell Berry, who wrote 25 years ago this week in the hauntingly prescient Thoughts in the Presence of Fear (in the wake of the September 11 attacks):

We did not anticipate anything like what has now happened. We did not foresee that all our sequence of innovations might be at once overridden by a greater one: the invention of a new kind of war…

We had accepted uncritically the belief that technology is only good; that it cannot serve evil as well as good; that it cannot serve our enemies as well as ourselves; that it cannot be used to destroy what is good, including our homelands and our lives.

Just as he warned us of what not to do, he also patiently pointed the path home: one that protects our lands and our lives. The antidote remains the same. I want to close by offering Berry’s first two principles of a just economy to guide us as we work to re-align with life (these invitations are core to what I think of as an Economics of Belonging):

1. Always ask of any proposed change or innovation: What will this do to our community? How will this affect our common wealth?

2. Always include local nature – the land, the water, the air, the native creatures – within the membership of the community.

Rest in power to a man who understood and embodied what it means to live in right relationship. Let us honor his legacy by preserving a future for life and the planet.

---

P.S. How I used AI in drafting this post.

With thanks to a commenter who asked, I felt inspired to add a note here about how I used AI in drafting this post (which is also how I’ve been using AI more broadly in my work). Generally, I seek support at four levels:

1. Research: I work with Claude on large-scale literature review tasks, e.g.: is there a generally accepted definition of Artificial General Intelligence (AGI)? What are the competing schools of thought? What are the different elements that together comprise “existential risk?” I spent a huge chunk of time trying to arrive at that 5-part definition that I ultimately settled on in the piece.

2. Honing my thesis: Whenever I write this is my biggest struggle. I often have multiple ideas that I’m trying to weave, and there’s a constant risk that I’ll overload the post and lose my readers. I felt clear that I wanted this to be an explainer, that I wanted to try to offer readers a sense of agency, and that I wanted to bring my own unique lens to the AI conversation… how to hold those potentially competing threads together? This is asking Claude (where I have a paid subscription) and ChapGPT (free version: I canceled my subscription when OpenAI signed its Pentagon partnership back in February) to help me clarify my own thinking/scope before I begin writing.

3. Clarity and structure: 2026 is the first year I’ve been using AI to support my writing, and this has been a huge help. Often the most time-consuming aspect of writing is the cutting/restructuring: do my arguments flow? Do premises support conclusions? Where am I likely to lose readers? This is uniquely hard for me with my neurodivergent brain: I have a hard time seeing where things that make perfect sense to me will not make sense to someone with a different brain. This looks like drafting the entire post (in my voice) then asking for critical feedback.

4. Word choice and quotes: This is the last touch. I’ll ask for help with word choice in service of clarity, and in landing on the right metaphors and quotes to support the piece. For example I asked Claude to search Pope Leo’s encyclical for the quotes most aligned with the arguments I was presenting (this is pure time-saving on my part, not a cognitively difficult task). And I asked for help figuring out how to co-hold the Moloch problem at the level of profit motive alongside the U.S.-China great rivalry problem.

In general all the writing/language is mine; I will adopt sentences and fragments that I like, and will rework my writing to improve flow. Sometimes I’ll adopt a sentence that doesn’t feel intuitive to me as a way to test my own comfort. This sentence for example: “At the geopolitical level in the context of U.S.-China rivalry, that same logic just wears a different uniform.” The first part of the sentence is mine; the second clause is Claude’s. I wouldn’t have said “wears a different uniform” but I thought the turn of phrase might help readers even if it sounded a little forced to me.

The main thing I’ve appreciated about AI: it’s a thought partner who both can meet me where I’m at — keep up with the wild complexity of my thoughts without losing the thread — AND help ground me in what other people will need to see. It helps me set my own foundation, then supports me in creating the scaffolding to invite others to see what I’m trying to convey. In that sense it’s been incredibly helpful, and helped me get to insights that I might not have reached on my own.

It’s also cumulatively a HUGE time-saver. A normal post in the pre-AI days probably took 40+ hours, the vast majority thinking, researching, and going down rabbit holes (maybe 4-8 hours actually writing/re-structuring, depending on the complexity of the topic). This post, which required a lot of research because I am NOT an AI expert, probably took closer to 20 hours of dedicated effort.

Thanks to Mari Amend for the question that prompted this postscript!