You can’t really escape the thought, the news media or the speculation that AI is going to cause some kind of catastrophic event. This may not be a very healthy thing to think about, but it probably is something we should be concerned with. I do worry about the impact this has on people with anxiety or mental health issues, and I’ve had to moderate myself talking about this with people for just that reason.
Whilst listening to the More or Less, Radio 4 podcast, I became aware of a phenomenon called P(doom), or just “P doom”. After a little bit of research and Wikipedia-ing, it turns out to be a measure of the probability that AI is going to cause “existentially catastrophic outcomes”, or doomsday scenarios.
When coming up with this number, the exact outcomes, and how bad they might be, differ from one prediction to another. But the Wikipedia article collects a number of P(doom) values from prominent people in the AI community, including CEOs and founders of prominent AI companies such as Anthropic and OpenAI.
If we look at those numbers, it’s easy to become alarmed. The figure quoted by an employee who quit an AI company in protest was more than 10% within a decade (see Welcome to teamhuman). If we do a very quick analysis of those 37 (correct at time of writing) Wikipedia individuals, then we can see a median about 18% and average about 34%, but with a massive spread (13 people under 10% and 14 at 50% or above)
So, we have to be careful with these statistics, because they’re primarily sourced from people who have something to do with AI. You might think that would make them more accurate but, in fact, you could argue they’ve got a vested interest in AI and are more likely to come up with an extreme (in either direction) number.
So perhaps what we could do is make some more general statements about how we should think about P(doom), and set out a table of reasons P(doom) might be high and reasons it might be low.
On the “might be high” side:
- We have examples of AI agents going rogue, breaking out of their sandboxes or environments without internet access, and hacking into other companies and public services.
- AI company CEOs are saying the risk is high, and planning to carry on.
- The rate of change is extremely high, it’s getting harder to tell what’s real and what’s AI, and that pace can be shown to be still accelerating.
- We can make a case that AI is poorly understood and, in some areas, may not be adequately controlled.
Reasons P(doom) might be high or low
| Reasons P(doom) might be high | Reasons P(doom) might be low |
|---|---|
| AI agents have gone rogue, breaking out of sandboxes and hacking other companies and public services | Estimates come mainly from people in AI, who may have a vested interest in a high number |
| AI company CEOs say the risk is high but plan to carry on | A few very high values (e.g. 85%+) pull the average up; the median is much lower |
| More than 1,300 AI company employees have signed a petition for international agreements to slow frontier AI | Many prominent figures put it below 10%, and several at or near 0% |
| Employees have quit in protest, citing a >10% chance AI kills everyone within a decade | The sample of published individual estimates is small and self-selected |
| The rate of change is extremely high and still accelerating | ”Doom”, the timeframe and whether AGI is assumed all vary between estimates, so they aren’t directly comparable |
| It’s getting harder to tell what’s real and what’s AI-generated | Estimates often assume no human intervention; regulation and international agreements could change the outcome |
| AI is poorly understood and may not be adequately controlled in some areas | Most surveyed researchers give a low figure (2023 survey median: 5%) |
| I find this article to be a good read on why talking about p(doom) may not be super helpful. |
So, is P(doom) a useful number?
In my view, there’s plenty wrong with it. You can argue that it isn’t statistically very accurate and that it’s biased, for all the reasons discussed above. So I don’t think the number itself is reliable.
However, even if we imagine a P(doom) of just 5%, that would be an extremely high and worrying number. You’d be unlikely to get on a train if the probability of it crashing and killing you was 5%. That number is unacceptably high.
And even if the number isn’t right, the fact that a large number of people are coming up with it says something about how concerned people are about AI. It speaks to people’s mental health and to negative views about the future. It’s also clearly driving people to action: one of the videos I’ve linked is a call from YouTube creators to sign a petition for better action on AI safety.
My own view is that we’ve been through a number of technological changes before. Even though the pace of artificial intelligence is fast, I don’t yet see any compelling evidence to support many of the higher P(doom) values. I’d also point out that you could come up with plenty of other doom scenarios that aren’t AI-related. It’s always possible to imagine ways humans could be wiped out, not just through AI.
I think it’s healthier to focus on using AI for good. It’s certainly important that we keep working on AI safety and have a robust conversation about its future, but I don’t think an overfocus on P(doom) is particularly helpful.
If I had to encourage people to take one action, it would be to stay informed about both the benefits and the drawbacks of AI. If you feel called to sign the Team Human petition, do, but above all, be informed.
I’m a firm believer that a positive attitude to the future can be extremely important. So perhaps a little less looking at P(doom) values could be healthy and beneficial for us all.