Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

Better Offline

Ed Zitron hosts Cal Newport and Adam Becker in a provocative discussion challenging the dominant narrative around AI existential risk, argui

Key takeaways

• The AI safety discourse is heavily influenced by a niche group of rationalists and effective altruists with religious-like conviction.
• Many leading AI researchers' concerns about existential risk stem from transhumanist beliefs dating back to 1990s sci-fi-inspired movements.

Main topics

• AI existential risk narrative
• Rationalist and effective altruist movements

Notable quotes

"They don't want to be human anymore. They want to use machines... to escape human bodies and solve the problem of death."

Conclusion

The episode argues that the dominant AI safety narrative is less about e

Transcript preview

Speaker 8 (0:00) This is an iHeart Podcast. Guaranteed human. Hi, Speaker 7 (0:04) this is Kylie Brakeman. Angela Geratana. Jeremy Speaker 8 (0:07) Colhane. Patrick McDonald. And Speaker 7 (0:08) we're the hosts of the Artists on Artists on Artists on Artists podcast. The Speaker 8 (0:12) improvised character comedy podcast where each week a new panel of artists discuss their craft. Join us for a panel of Eurovision participants. And we do a little Halloween sometimes. Does the Pope dress up? He winks on Halloween. He gets right up next to the microphone. Speaker 7 (0:28) I hear you can almost hear the eyelashes hit each other. Speaker 8 (0:31) You've never seen Italy so quiet as when the Pope gets up to wink on Halloween. Speaker 7 (0:36) Listen to Artists on Artists on Artists on Artists every Thursday Speaker 5 (0:40) on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I survived nine months in captivity, and I've spent my life exploring how other people survive what should have destroyed them. I'm Elizabeth Smart, and these are The Survivor Files. Every week, I'm with survivors who lived through the unthinkable, abducted, stalked, controlled, and nearly silenced. These are stories about what it takes to make it out alive. Listen to The Survivor Files with Elizabeth Smart on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. If Speaker 7 (1:16) your bookshelf and your For You page are equally important to your personality, welcome home. Pro Society is a weekly podcast that's part book club, part group chat for anyone who thinks Pride and Prejudice and Love Island deserve the same level of discourse. Each week, we're connecting the dots between books, the internet, and pop culture with your favorite writers, book talk creators, and plenty of overthought opinions. Yeah, I'm obsessed. I'm obsessed. Listen to Pro Society on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Speaker 5 (1:46) So like by age 11 or so, at least 70 % have a phone. Speaker 6 (1:50) I'm Dr. Joy, and this is Therapy for Black Girls. We spend our surviving middle school run getting into what's happening inside your kid's brain, their group chats, and their friendships, and what you can do about it. Speaker 8 (2:01) Your child is not doing this on purpose. Speaker 6 (2:04) That's the kind of perspective we've been pulling from the psychologists and psychiatrists who study this for a living. Listen to Therapy for Black Girls on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Speaker 6 (2:15) Self-care September is your reminder to slow down, check in with yourself, get off your phone, and maybe ask, why am I like this? I'm Gemma Spegg, host of the Psychology of Your 20s podcast, and this month we're helping you make sense of the person that you are becoming, from setting boundaries and navigating friendships to understanding anxiety. confidence, relationships, and attachment. Sometimes our biggest decisions, without us even realizing it, aren't driven by what we truly want, but by what will prevent us from feeling lonely or being alone. Listen to The Psychology of Your 20s on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Speaker 8 (2:57) The new NFL season is here, and you should be listening to NFL Daily as we march along to Super Bowl 61. It is in the name, NFL Daily. You'll have fresh content in your feed every day, all season long. That game-winning drive. Maybe it's Herbert. Maybe it's Geno. Maybe it's Mahomes. We'll have the highlights. If Fernando Mendoza or any of the rookies are balling out, we'll break down the tape. Join me, Greg Rosenthal, and an all-star cast of co-hosts as we preview and recap every game. Listen to NFL Daily. on the iHeartRadio app, Apple Podcasts, or wherever you Speaker 1 (3:30) get your podcasts. I'm Brett Coleman. And I'm Trevor Sykema. If you love football and want to understand what's really happening on the field, check out our show, NFL Pro. Every week, we break down the biggest plays, trends, and matchups using exclusive next-gen stats insights and NFL content you will not find anywhere else. Whether you're new to football, a fantasy football fanatic, or a diehard fan, NFL Pro will help you analyze the game like a pro. Listen to NFL Pro on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Speaker 5 (4:05) My name's Speaker 4 (4:07) Ed Citron, and this is Better Off Live. Speaker 4 (4:22) I am back and I'm announcing that we will be speeding up the process of researching powerful and dangerous podcasting technology. My researchers are spending over $200 million a day on GPU compute, trying to create and generate a recursive self-improving Zitron, and no amount of pressure from world governments will stop me. But before my greatest work is complete, I must continue to record. And today we've got an absolute barn burner. I'm joined by the claviculars of computer science, by which I mean Professor Cal Newport, an astrophysicist and author of more everything forever. Adam Becker, fellas, welcome back to the show. Thanks for having us. Speaker 3 (4:55) Yeah, I'm terrified of the idea of recursively self-improved. That means your newsletters will be 200,000 words long now? Is that what we'll expect? Yeah, they'll either be Speaker 4 (5:05) 200,000 words or 100 words long. Oh, maybe it's not the worst thing then. All right, keep going. When Speaker 3 (5:10) the Speaker 4 (5:10) perfect matrix of Zitron is complete. Well, today though... This was inspired by your wonderful piece, Cal, in the New York Times about the rationalists. And we are going to start very simple. Who are these people and why are they suddenly notable? Speaker 3 (5:26) Well, I'm probably going to actually throw the baton to Adam here in a second to give the history of this group because he and his book was one of my primary sources in trying to understand them. But let me just set the stage for why I was writing about them. I don't think people realize just how weird some of the lead figures are who are at, in particular, OpenAI, Anthropic, and DeepMind. And I don't think people realized that until last week when you had... a whole group of 20-somethings casually debating in their lunch hour whether it's 10 % or 50 % chance that we're all going to die within the next couple of years. And that, I think, caught a lot of people's attention. I'm like, wait a second. What is going on over there in those labs? Why is this type of rhetoric only coming out of those particular labs? Why does Jensen Wang think that this is ridiculous? Why is China laughing? at this narrative. There's something going on here that's not right. I think people are starting to wake up to this. And I think it really is this family of futurist philosophies that come out of the Bay Area that have mutated and split in various ways that are really defining our current moment and no one really realizes it. But Adam spent years, God help him, immersed in this world, has read everything they've written. He knows the history well. So actually, Adam, I'll throw the baton to you. Maybe you can give us a story of rationalism, And effective altruism and where they came from and how they've split it. To me, it's a fascinating, odd story. Speaker 2 (6:54) Oh, God. I mean. Speaker 2 (6:57) My standard response when people ask me this is, you know, well, I could write a whole book about it and I did. And, you know, I'm not going to get into the same level of detail I do in the book. But, like, basically, this is something that was spawned out of transhumanist message boards and email lists in the 1990s. And what is transhumanist? Yeah, that's a good question. Transhumanists... I guess the shortest good definition I could give you is transhumanists are people who don't want to be human anymore. They want to use machines and especially computers to basically escape human bodies and solve the problem of death. Okay. Yeah. So those guys. Yeah. Normal shit. Normal shit. Normal way to do. Yeah, exactly. Yeah. And and so, you know, these are people who are really excited about the prospect of AI, the prospect of, you know, nanotechnology that could. you know, basically serve as a kind of magic to do whatever they wanted. This was the 90s. Nanotech was like the hot thing. And it was a bunch of ideas that basically come from science fiction. Right. You know, they read too much sci-fi and confused it with reality and thought, you know, oh, well, because all of the sci-fi I read takes place in the future and all of the sci-fi I read has super intelligent machines in it. and nanotech and you know people uploading their brains into computers that must be what's going to happen in the future uh if we just you know work hard and wish hard enough for it Speaker 4 (8:48) and so just so i'm clear this is this is a certainty for them they're 100 sure this will happen they're Speaker 2 (8:56) 100 sure that they could make it happen right Speaker 4 (9:00) so it's in their hands yeah Speaker 2 (9:02) yeah yeah yeah like you know, they think, oh yeah, this is going to happen unless something, you know, ends civilization first or, you know, nobody listens to us or something like that. They think that they're working on the most important stuff in the history of humanity, basically. They think that they are going to, you know, make themselves immortal. Speaker 4 (9:23) And when you say working on, that seems like a fairly broad thing though, because a lot of these rationalist types don't actually work. on the stuff they work next to it yeah it's philosophy people yeah Speaker 2 (9:36) i mean it's not even so much that they're working next to it it's like they talk about it they like to talk about the prospect of nanotechnology they like to talk about uh you know super intelligent ai they like to talk about these things but but yeah so these people spent a lot of time doing that on message boards in the 90s and there was this one guy who showed up in the mid to late 90s on uh one of the most sort of influential of these message boards the extropians uh which you know don't get me started on that name it comes out of a misunderstanding that's Speaker 4 (10:10) a Speaker 2 (10:10) that's an annoying name no that's that's how you know this is Speaker 3 (10:14) a serious group of scientists that's right the extropians yeah Speaker 2 (10:17) that's that's correct yeah that's exactly right and they're like yeah we're gonna fight the second law of thermodynamics to which my response as a physicist is you know yeah good luck with that but Speaker 2 (10:31) basically this guy, this kid shows up named Eliezer Yudkowsky, who is a teenager at that point. Dropped out Speaker 3 (10:39) after middle school, dropped out of school. This is like when he dropped out. Speaker 2 (10:43) That is wild. Yep. Dropped out after eighth grade. That's correct. That's crazy, man. Yep. And. Speaker 2 (10:53) And yeah, Yudkowsky then goes on to write an enormous amount of, you know, his own deep thoughts on what he thinks the future of AI is going to be. And relatively quickly comes to the conclusion that AI is going to, you know, he takes an article of faith among these people, which is that AI is going to. become self-improving and lead to a super intelligent AI at some point in the relatively near future and lead to a singularity where everything changes and, you know, all of the stuff that the transhumanists believe is inevitable is going to happen. He comes to, Yudkowsky then comes to the conclusion that such an AI might not be friendly to humanity and probably wouldn't be unless it's done properly. And Speaker 4 (11:50) the singularity is when the computer is smarter than us, just so that I understand. Speaker 2 (11:55) It's really ill-defined. It's when the computer is smarter than us, or maybe it's when the computer can upload us into more computers so we can all be computer and everything is computer. The singularity is when... uh, is when a lot of tech, but all the time and everything, like it's not a well-defined thing. Um, Speaker 4 (12:17) yeah. And this is the thing they obsess over the thing that's ill-defined. Oh Speaker 2 (12:22) yeah. Yeah. Yeah. They, they think that that is pretty much inevitable unless, you know, we have like a nuclear apocalypse first or something. Speaker 3 (12:31) This is worth, let's just pause on this for a second. Right. I think it's really important because These ideas that I was talking about sound very familiar to what people are talking about today. They were talking about this. This is about the early 2000s is roughly when Yacalski kind of had this change of heart of superintelligence, of course, is inevitable, but now he was worried about it instead of being excited about it. This was all thought experiment. There's no LLMs. This was during an AI winter from the computer science perspective. It was entirely inductive thinking. directed by ideas from sci-fi. So they just sat there and thought about this, right? So these are not, this is not an engineer looking at a particular technical trajectory and saying, uh-oh, I'm worried where this is going to go. This is people just thinking through the thought experiments of, well, If AI could recursively self-improve, it would get super intelligent. What would it happen if you had a super intelligent AI? And then just sort of exhaustively working out in their own minds various scenarios about, well, maybe it would be friendly, but maybe it would trick us. And what if it did this or that? And so I think that's important. So you have all these ideas are established well before there's any particular technology that people are worried about, established entirely based off of, at this point, thought experiments. Speaker 2 (13:45) Yeah. So this isn't Speaker 4 (13:47) through proximity to actual scientists? Nope. So nothing about witnessing the actual technology following other than arm's reach? Speaker 2 (13:58) Yeah. No, I mean, it is exactly as Cal just said. And another way of phrasing that is it's essentially a kind of extremely dry fan fiction that they're engaging in. They are writing. Speaker 2 (14:14) you know, anxiety fueled scenarios based on these ideas that they got from science fiction, thinking that that's what the future is going to be. Do they believe it? Oh yeah. So they're sincere. They're just odd. Yes. Yeah. And at the time you can like dismiss them as sincere oddballs. What happens is, uh, Yudkowsky just keeps writing and writing. And gets invited to contribute to some group blogs and then spins off his own sort of blog called Less Wrong, which becomes a community hub for people. Yeah, that's him. But the other thing that happens is it's around this time, like the late 2000s, early 2010s, somewhere in there, that Yudkowsky meets Peter Thiel. Oh. And Peter Thiel gives Yudkowsky a bunch of money. How old is he at the time? Speaker 3 (15:09) Oh, we did this math. We did this math, Adam, with the fact checkers. Yeah. This is like 2000, the first decade of the 2000s. He was like mid upper twenties. Yeah. Something like that. Speaker 2 (15:21) Yeah. Yeah. That's right. Speaker 3 (15:22) Huh? It's like a decade of him, a decade. Just give in a Speaker 2 (15:26) child money. I mean, that's not out of character for Peter Thiel. But Speaker 3 (15:33) he literally pays people not to go to college, right? Exactly. The Speaker 4 (15:36) thing is, though, there's Speaker 2 (15:37) the Speaker 4 (15:37) Thiel Fellowship where it's like, I'll pay you not to go to college to build something. This is just like, I liked your blog, kiddo. Pretty much. You think the world's going to explode. You better blog about that some more. Speaker 3 (15:49) They met at, I'm trying to get this right, but they were at an event called the Singularity Summit. I believe Speaker 2 (15:56) that's where they met. Yeah. Or maybe that's where, it gets complicated. They definitely both attended. Yeah, it does. But a Speaker 3 (16:04) lot of money enters the scene, right? This is where a lot of money starts entering this world. Speaker 2 (16:09) Yes. Yeah. Yeah. And and Yadkowsky, along with some other, you know, transhumanist futurist types like Ray Kurzweil starts this, you know, set of conferences, the Singularity Summit and, you know, just gets people together every year to talk about this stuff. And. The community of rationalists on Less Wrong just, you know, start, you know, gets bigger and bigger and they start meeting up in the Bay Area. They start living together in group houses. What does rationalist mean? Rationalist basically means you spend a lot of your spare time reading Yudkowsky's writing. Yudkowsky thinks at this point that if you don't believe that the singularity is coming and that. AI is an existential threat to humanity unless you figure out how to do it right. He thinks if you don't believe those things, you're not thinking rationally. And so he decides to post a lot about how to think rationally. That's it? Yes, that's Speaker 4 (17:17) it. That's it? That's the entire rationality part? Yeah, that's why he writes... It's not about living rationally. It's just about their so-called rational belief. The computer might wake up. Are you fucking kidding? Speaker 3 (17:30) Well, they apply it to their whole lives. Yeah, exactly. That's Speaker 4 (17:33) why Speaker 3 (17:33) the two big blogs are called Overcoming Bias and Less Wrong. It's all about this sort of like really sort of tech obsessive way of I'm going to be super accurate in the way I think, right? And this is, I'm going to, I am not going to be overcome by cognitive biases. I'm not going to be tricked. And if we can just be. like hyper Sherlock Holmes is like, that's how we're going to unlock the potential of the world. But you Kowski thinks like the ultimate application of this is preventing the extinction of humanity by preventing super intelligent AI. And it like, that is why we have to raise an army of like hyper rationalist people. It's our only chance to survive the, the coming AI apocalypse. So like, it kind of becomes his obsession is that like this idea of like hyper rationalism that he's the leader of. is going to be, that's the John Connor of the Skynet scenario. Speaker 4 (18:26) Right, but they don't seem to be doing anything other than talking. Speaker 3 (18:31) Yeah, Speaker 4 (18:31) no, at this Speaker 2 (18:32) point, they're not. They're doing some Speaker 3 (18:34) things. I mean, they're in the polyamory. I mean, there's some other things going on. Speaker 2 (18:38) But like, to be very, very clear, Yudkowsky is anointing himself as an expert on... you know, the art of rationality and cognitive biases and how to think. And, you know, this is an entirely self-trained, self-taught guy whose experience of the world is pretty limited. And he is posting hundreds of thousands of words about this on Last Wrong and then starts writing as a... you know, a combination of artistic outlet and recruitment technique. He starts writing literal fan fiction, a Harry Potter fan fiction called Harry Potter and the Methods of Rationality. Oh. Yeah. It's like 600 Speaker 3 (19:27) ,000 words long or something. Yeah, it's longer than War and Peace. I'm sorry, Speaker 4 (19:30) even I would never write something that long. Even I. Speaker 4 (19:38) 200,000 words on anything is terrifying. I mean, this is 600 Speaker 2 (19:42) ,000 words. But what the fuck was it even about? Was it like... Oh, Speaker 4 (19:46) it Speaker 2 (19:46) was about Harry, who is, you know, I think fairly clearly an author self-insert of Yarkovsky. Harry being the specialist boy. And finding ways to change and save the wizarding world of Harry Potter using his special logic and science powers. Through logic and reason. Yeah, exactly. Yes. Right. Well, why? I guess it's just creative. Yeah. I mean, I interviewed him for my book and that was what he claimed. And I believe that it started out as a creative thing. But, you know, it was also very clearly him playing with these ideas that he has about, you know, this is the only way to think clearly. Speaker 2 (20:40) And it did end up becoming quite a recruitment tool for the rationalists. Speaker 3 (20:45) And just to put a – just make sure we have like the timeline beats clear and then Adam fact check me here. There's sort of this shift that happens where in the early 2000s – actually, I think his first money was – it was Peter Thiel, but it was also that other internet entrepreneurship couple. They fund something called like the Institute for – AI Singularity or the, what is it? Oh yeah, the Speaker 2 (21:04) Singularity Institute for AI. That was Speaker 3 (21:07) - Speaker 2 (21:07) Originally, yeah, originally the funding comes from this couple, the name of which I don't remember, but eventually, like a few years after that, in like the mid to late 2000s, he meets Thiel and Thiel gives them like real money. Speaker 3 (21:18) Yeah, and this is where, so as the 2000s move on, so that Singularity Institute was, super intelligence is going to be a beneficial godhead that's going to redeem history. Well Speaker 1 (21:29) - Speaker 3 (21:30) He shifts pretty, my understanding is like once that's a full-time job and he's thinking about it a lot, he shifts pretty quickly. He does that first big report. He shifts pretty quickly. Like, actually, wait, I don't think we'll control AI. We should not build it at all. It's going to kill us. Like there was this shift that happened like kind of quickly in that area and he shifted Valence and then eventually they renamed it to MIRA, MIRI, the Intelligent Research Institute. Speaker 2 (21:53) Yeah, that's correct. Yeah. So, so just, just to get the timeline straight here. Speaker 2 (22:00) Yudkowsky makes that shift from, you know, superintelligence is going to solve everything and we should be aiming for it as fast as possible, to, oh my god, it's going to kill us all. He makes that shift around the same time Singularity Institute for AI is founded, in the early 2000s. Right. Is there an antecedent, or does he just one day wake up? He's just thinking. He's just been on these message boards for a long time. And he also saw that like, you know, the the nanotech guys who wanted a lot of the same things and made a lot of the same promises back in the 80s and 90s, people like Eric Drexler. Speaker 2 (22:38) They had their own sort of institute. And Yudkowsky, you know, at this point, he's, you know, unemployed and about 20. And I think he's he's like, I want to work on this full time. Can I turn this into a job? And finds a way of doing that by, you know, co-founding this institute. But I also I want to I want to just push back on one thing that you said, Cal. OK, it's not. quite that, you know, Yudkowsky wants to stop AI at all costs. He wants to stop AI at all costs until he and others figure out how to do it safely, at which point he wants to go full speed ahead. He still