AI Researchers Are Panicking | What Comes Next is Worse Than Nuclear Bombs

Digital Disruption with Geoff Nielson

Kevin Roose discusses the high-stakes race toward Artificial General Intelligence (AGI), highlighting how personal rivalries and existential

Key takeaways

• The 'P-Doom' statistic represents the probability of AI causing human extinction, with some researchers estimating it as high enough as 75%.
• The AI race is driven not just by innovation but by personal animosity and mistrust between leaders like Sam Altman and Dario Amodei.

Main topics

• P-Doom and existential risk
• Personal rivalries in AI labs

Notable quotes

If you told me there's this plane that you're going to get on and it has a 10 % chance of crashing, I'm not getting on the plane.

Conclusion

The episode provides a sobering look at the high-stakes competition in

Transcript preview

Speaker 2 (0:01) There's this statistic that people in San Francisco like to throw around called P-Doom, which is your probability estimate of how likely we all are to die from AI. I usually say mine is about 10%, which in San Francisco qualifies you as something of an optimist. I meet many people at AI companies whose P-Dooms are 30, 40, 50%. I met one person the other day who said it was 75%. So, like, I am somewhat optimistic by the standards of people who are very freaked out about this stuff. But 10 % is not a small number to me. If you told me there's this plane that you're going to get on and it has a 10 % chance of crashing, I'm not getting on the plane. Speaker 1 (0:43) Why are the smartest people in AI continuing to build a technology they believe could end humanity? Kevin Roos spent years investigating that question, interviewing more than 150 people, including the CEOs leading the world's most powerful AI companies. What he uncovered is a race driven by innovation, fear, ego, personal rivalries, and the belief that if they don't build AGI first, someone worse will. Today, Kevin takes us inside that race. reveals what its leaders are saying privately and asks whether we may have already crossed the point of no return. I'm Jeff Nielsen, and this is Digital Disruption. Kevin, thanks so much for joining today. Really excited to jump into it. And maybe to start things off, I don't think to anyone's surprise, AGI, let's talk about it. I'm... particularly excited to have you here as a journalist, as an investigator. One of the things I find a bit troubling in this area is so much of the talk of AGI these days is coming out of the heads of frontier labs. And, you know, they're not exactly neutral in this conversation. They're fundraising. It's hard to take everything they say at face value. So I love talking to people actually doing the digging. So I'm curious. Based on that, based on your investigation, you know, what's kind of your perspective about AGI? What have you found? And how do you, you know, separate some of the noise from the signal? Speaker 2 (2:08) It's Speaker 1 (2:09) a Speaker 2 (2:09) great question because I think if you asked 100 people in AI what their definition of AGI is, you would probably get about 150 different answers. Even the same people sometimes disagree with themselves internally about what this means. Some people say this is a computer as smart as the average human across a lot of different tasks. Some people say this is an AI program that is smarter than every human at everything. So among the AI companies, there are various definitions that are at war with each other. For my purposes, I was more interested in studying this as a historical phenomenon because the notion of AGI is at this point more than two decades old. And for about the past decade, there have been a community of people who made it their mission to build AGI, whatever that meant to them. And that was the question I was trying to answer in this book was, who are these people? Why did they start doing this? And why did they persist in trying to build this very powerful AI system, even as many of them had serious reservations? And some of them even thought that this technology could lead to human extinction. So I spent... several years looking into OpenAI, Anthropic, Google DeepMind. I interviewed more than 150 people, including the CEOs of all those companies, and just tried to really archive the complete story of how we arrived at this moment, starting from before the large language model even existed. That is sort of the past decade. We've gone from having no large language models at all. That started in 2017. to now 2026, where we have models that are solving novel physics problems and committing autonomous cyber attacks and writing most of the code on the internet. So that was the question that I was trying to solve, was not like whether AGI is real or not. That is sort of beyond my scope. The question is, the people who thought it was real and who tried to build it, what was their Speaker 1 (4:19) story and how did that happen? Speaker 1 (4:23) It's a really interesting framing. And I think it's fair because there's a whole question about whether it's a marketing term, whether it's still relevant, what exactly it is, and trying to get any group of people to agree on that versus, to me, the underlying interesting story, which is the capabilities now, the people who have built it, their motivations, and where they see the world and where it's going. So maybe with the latter in mind, going back to this notion of... these labs, these leaders as natural fundraisers, as people who, you know, I would never go so far as to call them outright liars, but as people who have an agenda here, how, when you're a journalist, do you try and separate, you know, what they're saying on the record, what they're, you know, what's kind of marketing from, you know, what you see as underlying fact? Speaker 2 (5:16) Well, I try to talk to all of these people, both on and off the record. So I try to interview them. I've had all of the CEOs on my podcast. I've talked to them at length for this book. And I've also spoken to a lot of these people off the record in situations where they have really no incentive to lie to me. And I've also gone back and looked at some of the things these people were saying before they even had companies to promote or large language models to sell. So, for example, I think Dario Amadei, the CEO of Anthropic. gets accused a lot of kind of using these claims about AI safety to trump up sales of Claude or whatever the accusation is. But if you go back and look, I mean, he's a sort of primary character in the book because he was thinking and writing and talking about the same AI safety risks back in 2016. He had a paper that came out when he was a researcher at Google. called Concrete Problems in AI Safety. That's all about some of the same things we're seeing in reality today. Things like reward hacking, things like misalignment. These were some of the issues he was concerned about. At the time, there was no Anthropic. He worked at OpenAI, which had not built anything close. Actually, he was at Google when that paper came out. So he really was not building anything that could be considered modern AI. But these people have been pretty consistent over the years, both about their... sort of excitement and their fears. Sam Maltin used to also warn about the risks of what he called machine superintelligence, that it could kill everyone on earth. So, you know, take them with a grain of salt now that they have big companies to promote and they're trying to IPO, but also go back and look at what they said before any of that was true. Speaker 1 (7:07) Well, and it sounds like you're pushing back on some of these accusations and saying the drift that people are up in arms about is actually not nearly as bad as it's being made out to be. And that the consistency level with these people over time is actually quite high. Is that fair or is it more nuanced than that? Speaker 2 (7:27) I think it's a little more nuanced than that. Some of them have definitely changed their tune. You'll notice Sam Alton spends a lot more time talking these days about how excited he is about... the future of AI than he did back when he was starting OpenAI to try to prevent the apocalypse, essentially. And that was another fascinating thing that sort of came out in the reporting of this book was, you know, these people like to frame this as some grand sort of ideological or technological quest, but it's really a bitter personal feud between many of them. These guys all... hate and mistrust each other. Like Sam Altman and Elon Musk start OpenAI because they are worried that Demis Hassabis over at DeepMind, which has been acquired by Google at this point, is going to build the intelligence, the godlike super intelligence before anyone else. Then Dario Amadei and his Anthropic co-founders leave OpenAI in part because they don't trust them with this technology. So the whole story of the AI race is just a series of people growing mistrustful of each other and setting out on their own and deciding, well, I don't trust that guy to run the world. So I'm going to build my version of this first. Speaker 1 (8:44) If you're a technology leader, InfoTech is a name you need to know for getting ahead with AI. InfoTech helps leaders deliver on top technology initiatives. And one of our most popular programs right now is AI Strategy. We've helped hundreds of organizations build out an AI strategy and roadmap using our step-by-step methodology. Infotech's road-tested approach gives you a trusted blueprint to execute faster and avoid costly mistakes. And what you walk away with is a boardroom-ready AI strategy tailored to your needs and ready to execute. What sets Infotech apart is how we deliver. Unlike traditional research or AI tools, we don't stop at guidance. Infotech works with your team to build the strategy. You can do this through a focused one-week workshop, ongoing analyst support, or by using our full methodology and toolkit to get it done yourself. The outcome is an AI strategy ready for the boardroom and a team that's ready to deliver. To learn more about our AI strategy approach, check out the link below and don't forget to like and subscribe. Speaker 1 (9:50) I was talking much earlier in the year with Andy Mills, and he was saying something similar about, you know, you've got basically all the major AI founders who are a bunch of guys who at one point were in a WhatsApp group with each other. Like, that's how close-knit and personal this thing is. And I'm curious, because there's a couple of different ways you could look at this, depending on how generous you want to be. Like, to what degree is it, like, I think that... I can build the better technology or I think that that technology is bad for the world versus I don't like that guy. Screw that guy. I'm going to build something better. It's really Speaker 2 (10:30) both. I mean, you can't underestimate. I found when I was reporting this book, you know, many, many private sort of internal documents written by these people, Slack messages, emails, research memos. And it stands out to me that both they are concerned about these other people who are building, trying to build this technology. But there also is just a lot of personal animosity, right? Especially between like Dario and Sam. Those guys really, really do not like each other. And, you know, it's really interesting too. There's sort of this, there's this argument about moral responsibility that comes out in some of these conversations. Dario is an interesting case because I learned while reporting this book that his scientific hero, the person he has sort of patterned his career Speaker 1 (11:22) after, is Speaker 2 (11:24) this World War II scientist who sort of helps to invent what became the atomic bomb, Leo Szilard. He was a sort of reluctant participant in the Manhattan Project. He came up with the nuclear chain reaction. And then he tried to keep this a secret so that the Nazis couldn't discover it. And when it became clear that it was going to be impossible to keep this discovery a secret, he joined the Manhattan Project. And his logic was, we have to build it before the bad guys do or the world's going to end. And I think that's an instructive story because it really sits at the heart of why these companies are erasing. to build this technology, even though they know that like the atomic bomb, it could blow up the world. They think someone is going to build this. And if we don't do it first, it will be someone bad and untrustworthy. And so to stop that, we have to race ourselves. Speaker 1 (12:27) This Manhattan Project analogy is very much, I think, kind of a cornerstone of the findings you've had and the arguments and information you're presenting in the book. I'm curious if you can unpack that a little bit farther. I mean, you talk about it being a race, the kind of good versus evil framing. How does AI as a technology fit into that, I guess, at a deeper level versus just this is a world-changing thing? I mean, Speaker 2 (12:57) I think it's a pretty apt analogy. It might be the thing that is most analogous to AI, the atomic bomb, both because of its power. We're seeing now the effect that very powerful AI systems can have. They can go autonomously hack companies. They can develop malware. They can break into systems. The federal government and militaries around the... globe are worried about the strategic use of this technology in war fighting. And it is essentially a dual use technology, right? Splitting the atom could result in nuclear power, which I think many people believe would be good. And it can also produce a bomb, which is bad. AI can do lots of important and impressive things. It could cure disease. It can... make office workers more productive. It can teach people things as tutors, but it can also do really bad things. And so I think the dual use nature of it makes it, it's not a perfect analogy, right? There are fewer positive uses for the atomic bomb than for AI, but they both are technologies that either have or are currently kind of reshaping the global order and rippling through institutions and that people are being forced to respond to very quickly. Speaker 1 (14:23) I do really like the analogy. And just following the thread, it's interesting when you listen to some of these guys speak, how much, to your point, they talk about its potential as a weapon versus, you know, as an energy source and a force for good. I'm curious from your perspective, Kevin, being sort of the, you know, the investigator in all of this. Where you see it coming out? Is it truly both? Do you think we'll see it act as a force for both in the future? And is this framing apt in that sense? Speaker 1 (14:57) Maybe. Speaker 2 (14:58) I think that's the good outcome, right? The good outcome is that it turns out like nuclear weapons, which we did manage to contain through international treaties, through nonproliferation agreements, through monitoring. We have not had, thank God, a global nuclear war. Speaker 2 (15:19) But that was a very real possibility there for a while during the Cold War. We came quite close to that. And I think we're at this sort of Cold War-esque moment right now where we have