An Insider's Guide to AI Doom

Deep Questions with Cal Newport

Cal Newport breaks down the current AI harm debate into three major ideological camps: the Doomers, the Accelerationists, and the Centrists. He traces

Key takeaways

• Eliezer Yudkowsky founded the Rationalist movement after concluding superintelligent AI poses an existential threat despite its utopian potential.
• Effective Altruism, influenced by Oxford philosophers William MacAskill and Toby Ord, shifted focus toward long-termism and preventing human extinction via AI safety.

Main topics

• AI safety and alignment debates
• Rationalism and Effective Altruism philosophies

Notable quotes

"The conversation surrounding AI harms has really heated up in recent weeks, but it's also at the same time become really confusing and hard to follow."

Conclusion

Understanding the ideological map of AI discourse allows listeners to

Transcript preview

Speaker 1 (0:00) The conversation surrounding AI harms has really heated up in recent weeks, but it's also at the same time become really confusing and hard to follow. There's many different groups involved, and it can be difficult to try to figure out who belongs to what group, what group believes what, how do they feel about each other. I don't know how you can follow the current AI discussions about harms if you do not understand this landscape. of different players. So that's what I want to help you figure out today. This is a short episode I'm going to record where I'm going to lay out the three major categories of people involved in the AI harm debate. I'll tell you who's in each category, what they believe, and how they feel about each other. So if you are interested in the current discussion on this technology, you need to understand how this landscape is actually configured. All right, so let's get into it. Speaker 1 (1:01) All right, so let me give you two caveats before we get started. One, I'm going to make some mistakes here. I'm basically just winging this episode. And the boundaries between these different groups I'm going to talk about are a little bit porous. And there's some people that fall in between and some people that won't like the characterization. So I apologize in advance, but I believe the landscape I'm about to present you is more or less correct and will definitely help you better understand the current discourse on AI harms. My second caveat is I'm going to try to keep my personal opinion out of it. That's not the point of this episode. I really do want to just give you, to the best of my understanding, the current lay of the land in the sort of AI discourse that's happening at the moment. Okay, so let's get into this. I'm going to bring up my tablet here because this gets complicated enough that we need to actually probably see how these different groups connect. I'm going to start over here. By talking about the first of the three major groups that's involved right now in the current AI discourse. And we're going to call these the Doomer Group. Maybe I'll pluralize it. Let's call it the Doomers. All right. This is an important group in the current AI harm debate. Who is in this group? Well, I'm going to start by putting rats and EA down here. These two philosophical movements. give the intellectual underpinning for the group that I'm calling the Doomers. So let me give you the quick history on this. You have to go back to Silicon Valley futurist movements in the 90s into the 2000s. In particular, there is a movement called the Extropians that was futurist and very utopian. It believed that... Technology was advancing at such a rate that it was going to deliver us in the future a utopia, probably a transhumanist utopia, where we merge with the machines in such a way that we could have infinite life or whatever else we think is critical to our utopian visions. There's two main technological threads that the Extropians wove together. One was nanotechnology, which was just becoming big in the 80s and 90s when a lot of the extropian movement got started. They were influenced by a thinker named Eric Drexler who wrote about this for both academic and public audiences. So they thought nanomachines would be able to repair our bodies, reverse aging, reverse disease, etc. The other major technological thread that the extropians held on to was AI. They said maybe we're going to have... super intelligent AI that could act like essentially a benevolent digital deity. So this was going to be the other technological move that was going to deliver us this utopia. All right. So that's the Extropians. They're pretty positive. Then you have a particular Extropian. This was a young man named Eliezer Yudkowsky who dropped out of school after the eighth grade, was reading a lot of science fiction. found his way to the Extropian mailing list, got really into these ideas, was writing a lot for these email lists, eventually got funding to start his own sort of techno-utopian institute. He came to believe as time went on that though super intelligent AI could deliver a utopia, it could also just as easily kill us all. And that it was actually more eschatological than the Extropians realized. That superintelligence AI was inevitable, but it's not inevitable that it was going to deliver good things. And so he began to write and think quite a bit about what would be required to prevent superintelligent AI from killing us all and to help make sure that people understood this danger before we created this technology. Speaker 1 (4:49) So Yakowski's group became known as the Rationalist, right? So that's why I have rats written here under Dumerism. It's a shorthand for the Rationalist. All right, but I also wrote EA under there. That stands for Effective Altruist. Now, this is a movement you've also probably heard a lot about recently. This is a little bit complicated, but I'll summarize it really briefly. So the origins of the Effective Altruist is William McCaskill and Toby Ord, who are two philosophers at Oxford in the 2000-2010s. They were heavily influenced, among others, by the Princeton ethicist Peter Singer to apply logic and rationality to being altruistic. And so they're trying to be very effective at altruism. So early on, the effective altruists, for example, were really big on raiding charities, right? Because they figured you want to give your money to a charity that's going to do the most good. per dollar invested. So they worked on projects to help rate charities. They also had another campaign called GiveWell, which said, look, if you have a high paying job, the best thing you can do for the world is keep that job, but give a significant portion of your high income towards really effective philanthropy. So like these are the type of things the effective altruists were thinking about. But pretty soon they also became as a general movement. Very worried about super intelligent AI. There's lots of different points of influence between the rationalist and the effective altruist. A key one is Nick Bostrom. He was also a philosopher at Oxford, but he was also heavily involved on the Extropian mailing list in the 90s into the early 2000s. And he was very convinced, especially by Yudkowsky's style arguments, that super intelligent AI was inevitable. And without care was going to kill us all. And so he was one of several voices in effective altruism that moved the movement away from just thinking about effective giving and towards what was known as a long-termism view that said, wait, the most altruistic thing you could do is prevent the extinction of humanity. And so he started an institute at Oxford that looked at different existential risks. But over time, super intelligent AI killing us all became the dominant. risks they cared about. So we're in a world now where effective altruism is a very large movement. It's incredibly well-funded, mainly by tech centimillionaires and tech billionaires. It deals with lots of things. It's not only focused on super intelligent AI in the way that rationalism really is, but it's become a dominant theme. William McCaskill gave this well-known speech, I believe it was at Google, where