He Warned AI Could Destroy Us. Now The Industry Is Listening — ft. Nick Bostrom

Prof G Markets

In this episode of Prof G Markets, host Ed Elson speaks with philosopher Nick Bostrom about the existential risks posed by artificial intelligence.

Key takeaways

• AI researchers at leading labs genuinely believe there is a non-negligible risk—up to 10%—of AI causing human extinction by the end of the decade.

Main topics

• Existential risks of artificial intelligence
• AI safety and coordination among frontier labs

Notable quotes

"I'm a fretful optimist. I'm really excited about the upside, but also very concerned about the risk of getting it wrong."

Conclusion

While AI's potential for transformative good remains high, Bostrom warns that

Transcript preview

Speaker 5 (0:00) Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow, none of them talk to each other. That's where Odo comes in. An all-in-one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs, and best of all, no spreadsheets. Stop managing software and start managing your business with one unified system. Try for free today at odoo.com slash vox. That's O-D-O-O dot com slash vox. Speaker 6 (0:38) I'm Ina Garten. I have a new podcast called Happy Hour with Ina. Each week, I'm inviting a really interesting guest to join me for a drink and a fun conversation at my kitchen table in New York City. I'll be getting personal with actors, chefs, comedians, musicians, and writers I admire. So grab a snack, pull up a chair, and join us. You can watch by searching for Happy Hour with Ina on YouTube. or listen wherever you get your podcasts. Don't we all need to have more fun? Speaker 5 (1:15) I'm Mitch Purse, and this week on Confessions of an Elite Athlete, I'm sitting down with Matt Fries, goalkeeper for the U.S. men's national team and New York City FC. We discuss how to prepare for one of the biggest moments of your life. You can hear it all by listening to Confessions of an Elite Athlete on YouTube or wherever you get your podcasts. Speaker 1 (1:52) Welcome to Profiteer Markets. Last week, a former Anthropic researcher revealed that employees at both OpenAI and Anthropic believe that AI could, quote, kill us all by the end of the decade. His post quickly went viral and several other AI researchers came forward to say that they actually shared the same concerns. Then over the weekend, Anthropic CEO Dario Amadei published an essay calling for the industry to slow down. the development of AI models. Sam Altman said he agreed with Amadé and added that OpenAI will not be going public this year. There is now a growing debate over whether these fears are justified or overblown. So we wanted to hear from one of the people who has been studying this problem longer than perhaps anyone. His 2014 book, Superintelligence, helped shape how the world thinks about AI. It influenced many of today's AI leaders, including Elon Musk, Sam Altman, and Ilya Sitskiva, who even named his company after the concept. Our guest is one of the most influential philosophers of our time, and he is here to help us make sense of just how dangerous AI could become and whether the warnings we are hearing today deserve to be taken seriously. This is our conversation with Nick Bostrom. AI philosopher and best-selling author of Superintelligence and Deep Utopia. Nick, thank you so much for coming on the show. It really is an honor to have you, especially at this time where your research and your writing is so relevant. I guess we should start with the tweet that went viral, the Jacob Coxon tweet, the now former... anthropic researcher who said that the people building AI, quote, earnestly believe that it could kill us all by the end of the decade. He said, this is not a marketing stunt. Then the world seems to sort of blow up, or at least the global conversation blows up. Let's just start with your initial reactions to that tweet and how it has impacted the AI conversation. Well, let's see if we can try to make sense of this situation. Speaker 3 (4:00) It is a very... confusing and perplexing moment, I think, for humanity. We being sort of close to the potential birth of super intelligence. The idea that there could be significant risks associated with this, including existential risks, is quite widespread, I think, amongst people close to this technology and in the frontier labs. And I agree that it's not a marketing stunt. I think it's coming from a sincere place, a sense that we are getting in quite deep here and we should really pay attention to what is happening. Speaker 1 (4:40) Elon Musk is saying kind of two different things. On the one hand, he, well, I should say that he tweeted back in 2014 that he read your book and that he thought that AI is, quote, potentially more dangerous than nukes. And then he retweeted it quite recently. He even said that he agreed with Dario Amadei in terms of the size of the problem. But then he also said that he thinks that it might be a marketing stunt too. It's not totally clear where he stands on this. I just want to play you this clip of what he said. Here's the clip. Speaker 6 (5:15) It certainly is like some crazy 4D chess to say that there's whatever, 10 % chance of annihilating. humanity. But by the way, how much allocation would you like in our IPO? Speaker 1 (5:28) Do you think there are any merits to that argument? Speaker 3 (5:31) I think there is merit to the argument that there is an enormous upside as well as these risks. That's very much my view. I'm a sort of fretful optimist. Speaker 2 (5:40) I also think there is a lot maybe of 4D chess or attempts to kind of play this out and think strategically about different things that could unfold. I don't think it's a simple marketing ploy. I mean, it would be a rather Speaker 3 (5:54) strange tack to take if you were a big company planning to make an IPO to try to convince the world that your product should be regulated or banned or stopped or that it's so dangerous that it might destroy humanity. I think that message comes from a perception that this is... a really big deal. And in particular, the competitive dynamics are intense at the frontier of AI. One might think if we're going to develop this very powerful, potentially risky technology with many benefits, Speaker 2 (6:30) that it would be important to be able to be really careful when we're doing this so that if at some point the risks seem to be very imminent, we could, you know, take a few extra months maybe to do like extra safety work, test it carefully rather than... immediately cranking all the knobs up to 11. Maybe we'll do it a little bit incrementally and sort of Speaker 3 (6:50) see how things go. But if you're one of these frontier labs and you decide that you want to take an extra three, four months to fine-tune the safety on your models, you risk just immediately falling behind and becoming irrelevant. Like somebody else will then take the lead, be the one who pioneers AI. Maybe somebody who's less scrupulous, more willing to take risk. And so the action space is kind of constrained if you are acting unilaterally as one of these frontier labs, even assuming the best motivation. And Speaker 2 (7:20) so hence these calls for putting in place some mechanism that would allow for the possibility of coordination, like maybe a synchronized... slow down of the pace at some stage if that's necessary and or some safety standard that all the Speaker 3 (7:37) entities competing at the frontier would have to meet so that the race doesn't go to the least careful but like that we can sort of have an opportunity to try to make an extra effort on safety so I think that's like the core thought that is driving Speaker 1 (7:55) a lot of this if it isn't a marketing stunt And if it's coming from a genuine place, I mean, the quote by one of the current anthropic researchers was that most people at the company believe that this sort of apocalyptic scenario of killing all humans, that there is a 10 % chance that that could happen. So if we are to assume that these are genuine beliefs, genuine concerns, then the question becomes, are they right? Are they warranted, that level of concern and those probabilities? What do you think? Do you think that these are valid concerns? I Speaker 3 (8:38) think that seems quite reasonable. I mean, some people have even higher P dooms. What is less obvious is what exactly the implication of Speaker 2 (8:47) that is. Like one might, the first instinct, obviously, is if something has a 10 % or greater chance of... destroying the entire future, killing us all. Obviously, we don't want to do it. We want to shut it down. But we have to pause and reflect. First of all, if some competitors slow down, it doesn't mean we don't get superintelligence. It might be some other company gets it or Speaker 3 (9:10) maybe another nation. Obviously, there's a geopolitical race towards AI between the US and China. That's one dimension. Speaker 2 (9:17) Second, if there is a pause that lasts for a long time, if it's not done right, It might perversely increase the risk. That could then be a sort of buildup of massive amounts of compute that is not immediately used to create the maximum Speaker 3 (9:32) amount of intelligence. And then that is a kind of dry tinder so that when you finally lift the prohibition, then you have a sort of compute overhang that might mean we sort of get to radical superintelligence even more abruptly and quickly than would otherwise be the case. You could argue that... that would be more dangerous than sort of incrementing our way up there more gradually. Then we also have the fact that although super intelligence is a big risk, it's not the only big risk facing humanity. I think there are also other existential risks on the path ahead. For example, with advances we've seen in synthetic biology, even independent of AI, I think Speaker 2 (10:12) that is creating really concerning possibilities for designing new forms of infectious diseases and things that could destroy the ecosystem. And Speaker 3 (10:24) further ahead, we can think maybe one day there will be a nanotech revolution that would sort of be sort of biotech to the power of two. We remain Speaker 2 (10:35) under the cloud of large nuclear arsenals. I think we've got a little... complacent maybe from the fact that we survived the Cold War without Armageddon, but the risk is still very much there. And at any moment in time, Speaker 3 (10:47) that could be another sort of spiraling conflict between nuclear powers. More speculatively, even... The basic insanity of human civilization is not guaranteed to remain forever. Like we have new information technologies that allow new Speaker 2 (11:05) memetic phenomena. If we look back at history, there have been various times when destructive ideologies have persuaded large numbers of people and led to like calamities that could arise again, but maybe now on an even more global scale and sort of cemented into place with... these technologies we already have developed that could allow unprecedented forms of censorship and surveillance and so forth. And so it's not as if we have a choice between a zero-risk safe path and then a risky AI path, but there are sort of risks on both. And if we develop safe superintelligence, it could help us address a lot of the other risks. And then just Speaker 3 (11:44) one more point is also the benefits, which are sort of urgent as well. If AI could allow dramatic breakthroughs in medicine, for example, every year of delay means a lot of people dying that could have been saved if we had advanced more quickly. And so we wouldn't want to delay it, I think, longer than is really needed, but some slight slowdown or pacing, as the term in vogue, might... have sort of a high benefit during certain critical stages. So Speaker 1 (12:24) I want to return to what we do about this, how we regulate, how we build safe superintelligence, because I agree it's important. But I do just want to linger for a moment on your conception of the probability of catastrophic risk. You mentioned that those concerns have a 10 % chance of... catastrophe are not unreasonable. You mentioned that there are many other researchers that have even higher P-dooms, which is sort of the shorthand for the probability of some sort of catastrophic civilizational event. What is your P-doom? Yeah, I've sort of refrained Speaker 3 (13:05) from giving a numeric on that. I do think the risks are significant and should be taken seriously. Speaker 3 (13:15) And we just have a lot of uncertainty about how hard this basic problem is of aligning super-intelligent minds. It's a technical problem. We've never had to solve it before, and so we're going into this. And Speaker 2 (13:32) if we are lucky, it will try to be easy. We can just kind of fumble our way through and things will be okay. There's also a possibility that it's so hard that we are kind of doomed no matter what. Hopefully that's not the case. But then there's the intermediate possibility that it's a really difficult problem, but not totally infeasible. In those possible worlds where that intermediate level Speaker 3 (13:55) of difficulty is what we're facing, then it might make a huge difference if we like the degree to which we get our act together and really like do our best possible job at this. Speaker 1 (14:03) From an observer's perspective, Correct me if I'm wrong, but it sounds like 10 % is sort of in the ballpark of what you deem to be reasonable. I Speaker 3 (14:16) wouldn't necessarily over-anchor on 10%. That's what this guy was saying. I think it might also depend, if one really wanted Speaker 2 (14:25) to nail it down to a specific number, I think one would have to confront questions about what exactly counts as an existential catastrophe. So I think there are some scenarios which we clearly all agree Speaker 3 (14:35) are very bad. others that are very good, but then there might be situations Speaker 2 (14:39) where like the future is just strange. Like maybe the world is radically transformed in a way that means that a lot of the things we currently value Speaker 3 (14:50) no longer exist, but then there are new things, maybe new complex forms of digital life, some sort of Speaker 2 (14:57) continuation of human-like things, but transformed in a radical way. such that even if now we magically could sort of glimpse this future and go in with like a video camera and see exactly what the future looks like, we might still be uncertain how to evaluate that, whether to think basically this was a success or that was a total loss. So I think a significant part of the probability space is that something strange happens where something is lost, something is gained, and it's maybe partially subjective how you sort Speaker 3 (15:27) of tote up the positives and negatives. Speaker 1 (15:29) Yeah, this... aligns with my perspective, which I'd like to get your views on. But it sounds like, in your view, saying that there is a 10 % chance of catastrophe is simplifying the problem to a fault, because the reality is probably a lot more nuanced than that. Speaker 3 (15:49) Yes. In my worldview, there is also the additional complicating factor that I take the simulation hypothesis seriously. Like one of my earlier work was the simulation argument. Speaker 2 (16:03) And so then there is the additional question of how you would evaluate scenarios where something goes wrong, but we are in a simulation and maybe the simulation would have shut down anyway at some point, or maybe people in the simulation get continued in another simulation or uplifted. And so there's just like this vast... space of possible things to think through if one really wanted to sort of extract a number out of that, which is part of the reason for my reticence. What Speaker 1 (16:31) seems to be happening, and I'm not sure if you would agree with this, but it seems as though it's not necessarily a marketing stunt that, you know, are we going to say this thing, there's a 10 % probability, and that way we raise money. But it does seem that maybe what's happening is that these people at these companies feel that the world isn't taking this problem seriously enough. And so maybe we need to say something that is not necessarily hyperbolic, but maybe getting there, or maybe overly confident about what the path actually looks like, perhaps because they feel that at the moment we don't have enough attention, we don't have enough safety protocols in place, enough guardrails, and we need the world to kind of wake up. And in that sense, maybe mission accomplished because maybe the world is waking up to the dangers of what is happening. It Speaker 3 (17:24) is slowly waking up. I think it has a propensity to press the snooze button and kind of, yeah, and I think like... Probably up until this point, at least, if anything, I think there's been a tendency to downplay people's true view for fear of sounding kind of crazy. Until recently, if you were talking about sort of Speaker