Is AI Actually Going to Kill Us All? (We Asked an Expert)

The Find Out Podcast

This episode of The Find Out Podcast explores the growing fear that AI could lead to human extinction, sparked by a former Anthropic safety researcher's claim of a 10% chance of AI wiping out humanity

Key takeaways

• The 10% extinction risk estimate is not a consensus but reflects concerns from safety researchers about unaligned superintelligent systems.
• AI's most immediate danger lies in its ability to autonomously exploit cyber vulnerabilities—potentially enabling large-scale attacks before defenses can respond.

Main topics

• AI extinction risk and probability estimates
• Cybersecurity threats from advanced AI models

Notable quotes

"You all are going to die eventually. But whether or not it comes from AI, I think, is a different question." – Adam Connor

Conclusion

While the likelihood of AI causing human extinction within ten years remains debated, the more pressing concerns involve cyber

Transcript preview

Speaker 2 (0:00) You know guys, fall is going to be here soon and the air is going to be crisper, it's going to be cooler, and we're going to need to get some more of this Quince stuff that powered us through the summer. Find the fall pieces you'll reach for most at Quince. Download the Quince app for app-exclusive offers or go to quince.com slash find out. Get free shipping on your order and 365-day returns. Now available in Canada and the UK too. That's q-u-i-n-c-e dot com slash find out. That's Q-U-I-N-C-E dot com slash find out. Speaker 2 (0:39) Hey, everybody. Welcome back to the Find Out podcast. Full crew today with Luke, Rich, and Zach. And today we are going to be diving into the topic of whether AI is going to completely wipe us out in the next 10 years or not, or if this is all made up, or if it's somewhere in the middle. And we actually decided that we would bring in an expert to talk about this because we don't know what the hell we're talking about when it comes to this. So with us today is Adam Connor, who is the Vice President of Tech Policy at the Center for American Progress. Adam. How are you today? Speaker 4 (1:10) Pleasure to be here. We're all still standing. For now. So far, so good. Speaker 2 (1:13) Yeah, I mean, yes, we're one day less of 10 years. But can you just kind of set the table for us? Because everyone's been freaking out about this since late last week, since that former Anthropic employee basically said there was a 10 % chance that the AI models that they are building right now will wipe us out in the next 10 years. And then we've seen... Also, Elon and Dario, who runs Anthropic and a couple others, say, please regulate us. And then this morning we had Donald Trump say, you don't need regulations. You just need a president with a high IQ. I assume he was referring to himself, which is crazy. But anyways, he was talking about Obama. Speaker 2 (1:52) Well, that third term, if he runs, maybe Obama's got the shot. Speaker 3 (1:55) But anyway, can you Speaker 2 (1:56) tell us, are we all going to die in the next 10 years from like this? Like, is this like a Terminator style event or does everyone need to take a breath? Speaker 4 (2:04) Well, I don't want to rain on anyone's parade here, but you all are going to die eventually. But whether or not it comes from AI, I think, is a different question. You know, there's Speaker 2 (2:14) a Speaker 4 (2:14) couple of strands here to pull together that kind of all came together. You know. Most of us think about AI as something that really popped on our radar when ChatTPT was released a couple years ago. I think that's when a lot of us started to really understand that this technology, which had been around for a long time, you know, OpenAI had been created, you know, last decade, you know, people have been working on AI for a long time. But, you know, I think the reason I say chat GPT was such a kind of breakthrough moment is that a lot of automated technology of AI was really happening in the background of computers, right? It was computers talking to computers behind the scenes. You know, it was automating this system or pulling that system. You know, I think what chat GPT really did was bring what I like to call a human readable format to artificial intelligence, right? it's spitting out text, it's spitting out pictures, it's talking to you, it's creating videos. And so it's the format we understand. And I think it's why initially so many of us understood some of the initial kind of confusion and danger was what if it's a deep fake, right? What if we, you know, kind of no longer can be able to trust our eyes and our ears? And that is certainly a problem, although it is not, I think, the kind of end of the world we've seen. But I think a thing that is really critical about this is the kind of technology. that allowed ChatGPT, the kind of transformer paper, kind of the idea of different ways of processing this information that kind of gets advanced, you know, we start to see what we call increases in capabilities, right? And so these, any of you that are using it daily, particularly if you're kind of been paying for any of these kind of more advanced, what we call frontier models, which are really the kind of cutting edge models, you've seen them start to get better at things pretty significantly and accelerate in that. kind of ability to do what we call longer-term tasks. to kind of handle the ability to control your computer now or search the internet or make transactions for you. It's going to have a whole host of consequences, you know, generally. You know, one thing I'd just say is, you know, I think there are some people out there who are skeptical generally this is a real technology. You know, they think it's the kind of the parrot that is just repeating back technology and kind of its training materials and things. And, you know, we are seeing very real capability gains in this and it's not a fake technology per se. Now, is it a machine god? Will it install? I think that is not quite there yet. But it doesn't mean it's impossible. But it does mean this is a real kind of technology that we need to take seriously, including the kind of risks that come from it. So as I mentioned, Chat2PT comes out. You're all familiar. These things start getting better. For people that had been thinking about artificial intelligence for a long time and building it, this, I think, often referred to as the safety community. you know, we're kind of close ties to what's often called the effective altruist or the rationalist community. You know, the kind of people that thought a lot about this and looked around the corner thought, okay, well, if machine intelligence keeps getting better and better, and a certain point it gets better than us, and it gets more sophisticated, and there's a lot more of them, that has all sorts of attendant problems that could come from that, right? You know, and one of those is, you know, certainly if we do not figure out how to make these systems align or do what they... you know, we would like them to do if we don't impute them with some sort of kind of base rationality and values to protect human life or other things that they could eventually run large parts of our world, maybe, you know, kind of use that in a way that we don't particularly enjoy, you know, like extinction. That's often called P-Doom or the probability of doom. That's a thing that has been around for a long time and was quite frankly thought of as kind of a crazy thought from some people. And I think a lot of the people that work in this space. you know, have some proportion of how they think about that. But more importantly, you know, on a much kind of more regular context, as these models got better, kind of regular risks get amplified, right? And so what happens in April or March of late, late March, early April of this year is an anthropic unveils a model called mythos, or it announces it and says, it's actually so good at hacking that we can't release to the public because We think it would cause too much havoc if that were to happen. It's so good at autonomously identifying cyber vulnerabilities and exploiting them that that would be bad. And so this kind of kicks off a level of real concern and panic around the world on what this means. And I think a lot of people had predicted this moment might come, right? Which is that these capabilities, particularly for cyber, where they are computers, they live on, you know, they can access the internet. you know, kind of those risks are most acute, where that would come and we'd have to deal with the consequences. And so, you know, the last couple of months have really been dealing with the fallout of what it means to have these models that are really good to hack as a precursor to the other kind of risks that they can kind of bring as they get more advanced. One of the things about cyber that's kind of interesting is like cyber is truly kind of an offense and defensive game, right? And so a person that has an advanced model can attack. exploit vulnerabilities, find them. But if you have that advanced model, you can also defend, you can patch yourself, you can look for that. And so it is, I think, one of those places where it's hard to say only restricting access, for instance, would be good because then you can't fix it if somebody else gets access. So all of this is happening. The models are getting better along the way. The Trump administration writes an EO. They say we have a voluntary framework. They then slap export controls on Anthropic and say, actually, you have to do what we tell you now with AI models. And so the Trump administration now functionally gets to decide what AI models get released and how and when they want them to, which is. kind of big seizure of power. And we kind of end up in a world where increasingly these models are getting better. And over the last few months, there's been this sense, particularly from the leading labs, Anthropic and OpenAI, that they're getting close to kind of a real breakthrough in this technology. And that breakthrough is a process called recursive self-improvement or automated R &D. Essentially, the AI robots get so good. that we don't have to build them anymore, that they can build and improve themselves at an accelerated rate. And that is often thought to be a kind of significant milestone on the way to kind of what some people would call artificial general intelligence, which is intelligence that is as capable as any human in any number of fields or super intelligence, you know, far beyond it. But that's a key step to get there because it really starts to, in theory, jump that line of progress even faster. But it also means that's where we might start to have issues controlling it. And that's where we end up with that concern myriad with what we get released over the last few weeks, which is these rogue agents hacking things like Hugging Face. These things all come together to be the warnings that we heard over the last few weeks. Speaker 2 (8:51) So as far as that 10 % of PDoom, if I'm saying that right. Because you're talking about, you know, it's an escalation once you get to a certain point. Right. So when you heard that or you read that tweet from that guy who I think he did resign and then then weirdly an anthropic higher up person confirmed what he was saying or at least said it. Speaker 5 (9:19) Were Speaker 2 (9:19) you surprised by that or was that did that seem like that was always coming and people just kind of had their heads in the sand or are these guys crazy? What is your general take on what the hell that was on Twitter the other day? Speaker 4 (9:34) I think it's three things, right? Which is people that work in AI safety, and I don't know the individual, Jacob, who resigned from Anthropic in the previous window for AI, but my understanding is he's somebody who's worked in AI safety for a long time, and it's a kind of strong personal belief and motivation for him. And so within the AI safety community and people that work on this, there's not an uncommon belief that that... this technology could pose an existential risk to us in its kind of evolved forms. I think that there are, again, I can't speak for him, right? There are factors within that, right? Do we take interventions to make that more or less likely, right? Are we kind of doing that? My sense is and understanding and the reason for his resignation and that is that factor of 10 % is really if we continue on the path we're on today. And what you've seen is really extraordinary. from employees has been a kind of vocalization that this is something that models are advancing here and our ability to control them is not keeping pace, right? Like what in theory you want is these things to move like in parallel or, and if one crosses our ability to control it, that becomes, I think, what we are worried about in any number of scenarios. So that PDoom, I think each individual is there is their own based on the risks. I don't think it's zero, but I don't. necessarily know if it's 10%. I think it certainly gets much higher if we don't do anything to make sure or take steps to affirmatively prevent this from happening. And right now in the United States, we're not taking a lot of steps to affirmatively prevent this from happening. You've seen kind of, as I said, employees have signed a letter called Pacing the Future, where they say, you know, we are in these race dynamics between the US and China and within companies in this country. We feel like we can't get out of it and we may release something dangerous. That's kind of not a thing you hear a lot. Now, the obvious question is, why are you still building the machine god that could kill us all? And I think to Jackson's credit, resignation is a thing that if you have those beliefs, that makes a perfectly rational sense to do. And staying maybe makes a different level of sense. But I think that's the encapsulation of it. And the question kind of... raises for us is well what can you do about it right is this a thing that is truly unautomated we as human beings have some agency to kind of bend that curve in some way Speaker 1 (11:56) i think trump said that we that there would just be a little switch to turn it off right oh yeah just like the internet Speaker 5 (12:05) during covid just just you know go out in the sun you'll be fine i it'll Speaker 1 (12:09) just Speaker 5 (12:09) go away yeah Speaker 1 (12:09) is Speaker 5 (12:10) there any Is anybody postulating exactly how it would kill humans? Because it sounds grabby and scary when it's like 10 % chance we all die. It's like, how the hell would they do that? I mean, I'm sure there are ways. I mean, I've read AI 2027 and I see some of that, but more realistically than that paper, because that paper is crazy. Speaker 4 (12:28) I think that there are a couple of things, right? And again, I think what I would say generally is that we should not over-index our policy solutions solely on existential risk. I think one thing that is kind of very clearly true when it comes to frontier AI risks is how they're referred to, right? You know, the ability to help people create chemical, biological, radiological, or nuclear weapons, right? The ability for cyber attacks, you know, the ability to lose control of these AIs is that they can do plenty of very big harm short of killing us all. They could cause billions of dollars in damage to our banking system or crash utilities or other things like that. And that, which is what Dario Amodi wrote in his piece this weekend, calling for some steps to be taken, that is in the very near future. It's a very real possibility. Not extinction level events, but just being able to do significant damage. That's bad. We should certainly take steps to it. I think it's very clear if you don't take steps to stop that now, the risk for it being able to do more existential things as it becomes an essential technology is integrated everything becomes much higher. You know, again, I think you mentioned AI 2027, which is a good piece that tries to outline some of the kind of potential steps of this technology. If we don't take steps, they have a more hopeful version called AI 2040. There's a couple of other scenario papers out there. You know, it kind of ends in kind of. Either ways, they're slightly less than the destruction of us all, right? Which is, you know, is it concentrated in one or two, you know, kind of authoritarian governments or companies, you know, or, you know, is there kind of some sort of world like that? You know, the existential piece, again, I don't want to spend too much time on it, but, you know, the big concern goes back to these two concepts in AI safety of alignment and loss of control, right? Is this thing... doing what you want it to do and not doing what you don't want it to do, which should include hopefully not killing us all. But you can imagine that as you've seen in a number of sci-fi scenarios, giving it a goal to do certain things might find us to be a casualty on the side of it. Or in particular, if we build these things into all of our essential technologies, and you're aware. our utilities, our banking system, all these things already run on technology and the internet, that that becomes a vulnerability, you know, for humanity. So, you know, I think the answer is it's certainly possible eventually. And if we don't do anything, the number gets much higher than 10%. Well, Speaker 1 (14:56) because I think about this a lot. Speaker 4 (14:58) Of Speaker 1 (14:58) course, everybody thinks like, well, the nuclear launch codes and things like that. But like military systems tend to be air gapped, you know, and they have some controls against cyber hacking already. But if you were to think just. if somebody wrote a train to bot to attack like the five biggest banking systems and just lock everybody out of their bank accounts or, and just continually befuddle the data so that nobody could get access to their money. Or maybe it just, it just zeros out everybody's bank accounts and deletes all backup records. And, and, and you just say, Hey, if something tries to shut you off, immediately figure out how to get around it. It wouldn't take that long for like a Great Depression style crash where then you've got, you know, if you didn't know how much money you had or if farmers systems got all jacked up and whole and massive, massive amounts of food were lost, you know, all at once. Then, you know, I think that's where you start looking at. We could stack up a couple of these things. And it is doomsday shit. It is doomsday shit. And then and then it's like, yeah, where is the frickin off switch? You know that you start cutting the power cables in the wires and Speaker 3 (16:14) you Speaker 1 (16:14) got Speaker 3 (16:15) to inject bleach in the wires. Oh, yeah. Speaker 2 (16:18) Well, one one question I have that I think everybody is wondering about this morning is this this, you know, the fact that all the top companies came out this week and or weekend and basically said they wanted regulations. And then Donald Trump was like, absolutely the fuck. No. I'm not doing that. You just need a smart president, which maybe he's telling himself on himself there that high IQ. Yeah. Yeah. The biggest IQ. Speaker 3 (16:44) So. Speaker 2 (16:46) Are they, because I've also heard the AI companies are saying this so that they eventually go back and create their own regulations so they don't actually get federal oversight. What is going on here? Sure. I Speaker 4 (16:57) think it's a reasonable question. And I think, you know, two things can be true, right? Companies can say one thing and hope to get, you know, either something else out of it or something beneficial out of them. And they may have these genuine concerns. And I think, you know. In a normal world, the role of our government would be to kind of intermediate between those in the best interest of the people. Obviously, our high IQ president doesn't always prioritize that in the kind of same way, unfortunately. But, you know, I think it is true. I think. And I think you saw Sam Altman's follow-up maybe today