A.I. Safety Goes Mainstream + a ‘Hard Fork’ Exit AMA

Hard Fork

This final episode of 'Hard Fork' explores the sudden mainstream attention on AI safety, featuring reflections from hosts Casey Newton and K

Key takeaways

• AI safety has moved from niche concern to mainstream discourse, with even non-experts now asking if AI could lead to human extinction.
• Former OpenAI and Anthropic researchers like Jacob Coxon have publicly criticized the industry's reckless pace toward self-improving AI.

Main topics

• AI safety becoming mainstream
• Insider resignations from OpenAI and Anthropic

Notable quotes

"I think there's a greater than 10% chance that this could happen." – Evan Hubinger on AI extinction risk.

Conclusion

As 'Hard Fork' concludes, it leaves behind a legacy of early warnings about AI

Transcript preview

Speaker 3 (0:00) I'm Winna Liu. I write the game Connections, one of the puzzles from New York Times games. And I love horror movies. I love my dog. And I love trying to trick you. Speaker 8 (0:09) I'm Tracy Bennett. I get to pick the wordle word every day, which is not as easy as it sounds. A fun fact about me is that I am descended from a witch who was put on trial in Salem. New York Times games are made by people, like the ones you just heard from. Go to nytimes.com slash games to start playing today. Speaker 7 (0:30) This is actually really big. Do you know what I did? ahead of our final episode. I have to say, I'm actually quite proud about this. What'd you do? I finished Speaker 6 (0:37) reading your book. Speaker 7 (0:38) Wow. And I really loved it. The AGI Chronicles. It's very fast moving. It's fun to read. And something that I really liked, it was gossipy and juicy and bitchy in a way that these business books almost never are. And of course, I don't want Speaker 5 (0:53) to Speaker 7 (0:53) take any credit for anything that you did, but I would like to think that just a little bit of me rubbed off on you. Although the truth is you've actually always been very gossipy ever since I met you. Speaker 6 (1:01) I know. Well, I did not set out to write a juicy gossip book. It was supposed to be like a sober history of science. And then you start talking to these people, and they're the biggest gossips in the world, and they all hate each other, and there's so much drama. And I felt like I was freaking Harvey Levin on TMZ just sitting there scribbling all of this down. It's honestly very hard to write a sober history about people who are on mushrooms all the time. Speaker 7 (1:26) That's just not... really going to happen. Hey, let's be fair. They Speaker 6 (1:30) also do ketamine. Speaker 7 (1:31) Yeah. Okay. I'm sorry not to leave out the K enjoyers out there. Speaker 6 (1:35) Well, thank you for reading the book. I did not ask you to do that. I frankly did not expect you to do that. I know it's been many years since you've read a book, probably since Harry Potter, the last Harry Potter came out. But I'm honored. I'm touched. I'm so glad you like it. It Speaker 7 (1:51) was great. I can't remember. We were talking about books recently, and I mentioned that I was reading your book, and I mentioned some future book that I might read. And you stopped me and you said, Casey, my book is the last book you're ever Speaker 6 (2:02) going to read. And I thought, well, maybe Speaker 7 (2:07) I have something there. Speaker 7 (2:13) I'm Kevin Roos, a tech columnist at the New York Times. I'm Casey Newton from Platformer. And this is Hard Fork. This week, why everyone is talking about AI safety. And then for our final episode, we'll answer all your questions about the show and what we're doing next. I was thinking about getting lunch. Speaker 6 (2:31) Let's do it. Speaker 7 (2:43) Well, Casey, we picked a hell of a week to take off last week. Boy, did we. And thanks to all the listeners who wrote in asking us how dare we take any time away during a very busy and important time in the world of technology. Speaker 6 (2:56) Yeah, and to those listeners, we have one message, which is this is our last show. We heard you. We hear your frustrations. And so we are stepping down in shame. One star Apple podcast reviewers, your Speaker 7 (3:07) words have finally paid off. This will be the last you ever have to deal with the likes of us. Speaker 6 (3:13) So there has been just an enormous amount of news in AI over the past week or so. It seems like this conversation about AI doom, AI risk, existential threats from AI has really gone from sort of... our bubble to a much larger bubble that includes politicians and just random people in my life who have been texting me asking things like, are we all going to die for real? So let's talk about all of this. It has been one of the fastest, most sudden shifts I've ever experienced. Speaker 7 (3:45) Yes. And I hope if you're a longtime Hard Fork listener, you've been feeling somewhat vindicated over the past few weeks because I know Kevin and I have, right? Like we have been trying to bring you stories about this almost ever since we started the show nearly four years ago. And just within the past couple of weeks, it seems like there was some kind of inflection point and the vibe has shifted toward AI safety. Speaker 6 (4:06) Totally. So let's talk about what has been going on and what we feel about it. But first, since this is a conversation about AI, let's do our disclosures one last time. One Speaker 7 (4:16) last time, Speaker 6 (4:16) for old time's sake. Kevin? For the next 48 hours, I work for the New York Times, which is suing OpenAI, Microsoft, and Perplexity. Speaker 7 (4:27) And my fiance works at Anthropic. Okay, Speaker 6 (4:30) so let's kind of do a speed run through the AI news of the last two weeks. The first sort of domino that I think really started to fall here was the now viral resignation posts of Jacob Coxon, a former researcher at Anthropic and OpenAI, who went online last week and publicly announced that he was stepping down, saying that... Neither company, OpenAI or Anthropic, is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. This kicked off a firestorm of attention, follow-on posts from other AI insiders saying, we agree with him, people piling on and saying, this guy is crazy. But people basically, from what I saw, were saying, yeah, this is what we've all been talking about for years now. And in one group chat of Anthropic employees seen by The New York Times, one employee wrote that Jacob took the group chat public, and that's a good thing. Speaker 7 (5:33) Yeah, I mean, it was so fascinating to watch his response. And I have to tell you, this was a moment where I felt like, oh, no, I really am in a bubble in a bad way, because when I read his post, it did not faze me. Like this, like what Jacob said, as important as it is, is also like the median dinner table conversation in many households across San Francisco, including mine on many evenings. And so when I first saw it, I was like, oh, OK, he's decided to opt out. What I didn't count on, I think, Kevin, was that the response to that OpenAI hugging face incident has really gotten people thinking differently about these sort of things. And so I think people are just much more prepared to receive this message today than they were even a few weeks ago. Speaker 6 (6:17) Totally. And I had the same realization about the bubble that we live in here in the Bay Area when I saw a follow-up post from one of Jacob Coxon's former co-workers at Anthropic, Evan Hubinger. who sort of, you know, quote tweeted his post and said, I also believe that AI could kill all humans. And he attached a specific probability to it. He said, I think there's a greater than 10 % chance that this could happen. And my reaction to that was, that's it? Like, for me, in the world that we travel in, people who have a P. doom of 10 % are considered somewhat optimistic. Like, among the AI researchers at the labs that I've been talking to and interviewing, People have much higher probabilities that all this could go badly. And yet that number seemed to register on a scale that I had not fully appreciated. People are saying, wait a minute, these people are racing to build something that they think could have a 10 % or more chance of killing us all. Why are they doing that? They should stop doing that. Speaker 7 (7:18) Yeah, and I think maybe it's worth pausing for a beat, Kevin, on some of the logic here, not just in Jacob's post, but in some of the follow-on posts that you mentioned, because I think if maybe you just saw that post and didn't dig any deeper, you might be inclined to dismiss this as just more AI doom from a bunch of cultists. But I mentioned that OpenAI hugging face attack a little bit ago, and I think it's really important here, because as you heard Ajay Akotra say on... a recent episode, there was something surprising about how misaligned these AI systems are, even in their current state, which the researchers still see even today as somewhat primitive compared to where we think they are going to be in six months or a year, right? And what I heard Ajaya say was, look, we sort of assume we would start to see this kind of like scary collaboration between agents sacrificing themselves from each other. We thought that might come, but we thought it was going to come much later. So that's sort of like pillar one, I think, of what's going on here. Pillar two is all of the big labs, including OpenAI and Anthropic, they are racing toward recursive self-improvement. And they're putting out posts that are trying to say in a very gentle way, uh... the new models look like they're kind of getting there. We're already using these models to build the next version of the models. Like if you believe that recursive self-improvement is a spectrum, we're kind of getting ever closer to like the genuine article. And so if you have this unsolved alignment problem and you have imminent recursive self-improvement, then you might actually have a problem. And so I think it totally makes sense that some people would quit over it. Speaker 6 (8:55) Totally. And I think the vibe has also shifted inside these companies where like until maybe. three or four months ago, there were teams at these companies of safety researchers, alignment researchers, who were very worried about the risks. But there were also teams that were not as worried about the risks. There were teams that just worked on improving the capabilities of the models or researching new breakthroughs who were sort of like at odds with the safety researchers even in their own companies. And what I think we've seen over the past few weeks is that that has shifted, where now even the people who are sort of the optimists, the accelerationists inside these companies, the people working on capabilities, are getting spooked by how quickly the systems are improving. Speaker 7 (9:37) Absolutely. Now, should we talk about the reaction from the leaders at the big labs in the wake of all of these discussions that were kicked off? Speaker 6 (9:45) Yes. So on Saturday, Anthropix CEO Dario Amadei... released a 3,800-word essay called We Must Pace the Frontier, where he called for essentially a global coordinated AI slowdown. I'll just read a few lines. He says, We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. He proposes a few different ways to actually conduct this sort of coordinated slowdown. One of them is to put what he called embedded evaluators inside the major AI companies. He said that Anthropic was going to do this and not wait for the rest of the industry or regulators to force them to. So they are going to have essentially meter- like groups who are actually given employee-level access to their internal systems, who can kind of sit in their office and monitor their systems on an ongoing basis for signs of dangerous capabilities or mischief in the models. Speaker 7 (10:48) Yeah, and this is based on something that we already do at banks, for example, and I think it is a great idea. And notably, other leaders came forward and said that essentially they were going to follow suit. And among them was Sam Altman from OpenAI, who famously... does not have a good relationship with Dario Amadei. No, these dudes hate each other. They never agree on anything. And yet, Altman, to his credit, came out and said, we are going to do the same thing. Speaker 6 (11:11) Yes, this to me was the part that felt like, oh, we are in a new era of this stuff when even Sam and Dario can agree on the need for a coordinated global slowdown. You know that things have gotten quite serious. But also Elon Musk, who is engaged in a protracted legal battle with OpenAI and who does not... you know, have a great history with Dario and Anthropic either. And Speaker 7 (11:32) also, I would just say, he's not a big platform safety guy Speaker 6 (11:34) in general. Speaker 7 (11:35) Yeah, right. Speaker 6 (11:36) He also sort of co-signed Dario's statement. Demis Hassabis from Google DeepMind said that he agreed with the need for a coordinated global slowdown, too. So what we saw, essentially, is all the leaders of the sort of frontier or frontier-ish AI companies coming together to say, in more or less unison, things are moving too fast. We need to slow down. And we not only need to slow down ourselves, but we need to be slowed down. Some government or coalition of governments needs to put in place a coordination mechanism whereby we can sort of all press the big red button to slow down together. Speaker 7 (12:13) Yes. Now, unfortunately, though, we live in a time when, for very understandable reasons, no one trusts the tech companies to have our best interests at heart. Everyone suspects them of doing constant marketing hype. And we have a government that is relying on the AI industry to prop up the entire American economy. And so... The world that I hoped that we might have lived in, which is a bunch of lawmakers coming forward and saying, hey, guys, thanks for sounding these alarms. We're going to get right to work on some regulations that will prevent you from building an out-of-control AI system. Instead, we got a bunch of leaders, starting with President Trump, who essentially said all of this is ridiculous. So, Speaker 6 (12:54) yes, the Trump administration and many leading Republicans are very skeptical of all this doom talk. But there have been people who have rallied to try to do some sort of regulation in the wake of these statements. Politico reported that there are more than 100 Democrats urging. Republican House leaders to keep them in session until they pass some meaningful bipartisan AI safeguards. There are other members of Congress, including some Republicans, who have been taking this quite seriously. So I don't think it's quite as polarized as you're implying that it is. You're right. It is generally Democrats who are more worried about the AI safety concerns and Republicans in the Trump orbit who are less concerned about it. Speaker 7 (13:39) Yeah. And the Times had a great story this week about all of the proposals to regulate AI that have been introduced into Congress. Right. It is the case that members of Congress are introducing bills. It's just also the case that we have a Congress that very rarely, if ever, passes any sort of technology related regulation. And when the president has already sort of come out and said, we don't need any of this, the. Odds of a veto seem quite high. Speaker 6 (14:02) Yeah. So now we're in this moment where you have the AI company leaders calling for a coordinated slowdown. The public is really freaked out about this. I mean, you know, just to take, you know, my own experience as a proxy, like I have been just inundated with. you know, requests to go on TV and radio and friends and relatives and people I worked with two jobs ago texting me and saying, like, what is going on here? Should we be worried? I do think this has broken through in a way that other kind of AI risk stories have not. Even the Hugging Face incident on its own did not seem to reach this level of concern. So it feels like something is swelling here. And I think it's going to be tough for... President Trump and people in his orbit to maintain that there is nothing to worry about here when so much of the country, including many Republican voters, are concerned. Speaker 7 (14:59) Yeah. From the administration that brought you don't worry about COVID-19 comes don't worry about the rogue agent swarms. Oh, I wonder what happens next, Kevin. Speaker 6 (15:08) So, Casey, what do you think about this moment? Are you sort of skeptical? Are you? feeling more optimistic now that people are talking about this more? Like, where has your head been in the week since Jacob Coxon resigned? Speaker 7 (15:21) Well, listen, if you're a journalist who has opinions online, one of the best things that can happen to you is that you say something many years ago, and then eventually the rest of the world starts to agree with you. And this is sort of what I think is happening right now, right? We have been talking for years about the fact that these systems were developing very quickly, that the alignment problem was unsolved, and that eventually we might just get to some threshold where we started to see catastrophes in the real world. As we've said before, we are very lucky that this hugging face attack that has so captivated the world. ultimately did not result in a lot of harm. It didn't physically harm anyone. And I think hugging face the company is going to be perfectly fine. But it did start to wake people up to so many of the dangers that we are describing. So I think the question is, are we able to get from this current moment to some real safeguards here in the United States? Or will it just unfortunately take something even worse happening to finally get the elected officials of this country to get a law passed? I have to say, I have been feeling unexpectedly optimistic Speaker 6 (16:22) in the last couple of weeks. I think around the time of the Hugging Face hack, my worry about AI was really climbing. I was really feeling like we might be screwed. It might be too late to rein in these recursively self-improving agent swarms that are going out and creating these persistent botnets and doing all kinds of malicious things. And I'm still worried about that. But I think this sort of call for a slowdown, especially among these rival AI leaders who most of the time do not like each other and agree with each other on much and are racing to build this technology. I think the fact that they are now willing to say in public what many of them have been saying in private for years is a good thing. I think it gives the permission for their own employees to start speaking out about their concerns. I also think it gives lawmakers permission to take this more seriously because it is rare that an industry comes to Washington and says, please slow us down. We are out of control. We need you to regulate us before it is too late. I Speaker 7 (17:26) really want to draw people's attention to that because this gets at the marketing hype conversation, which is just constant everywhere I look online. It is costing these people something to say this, right? It is costing their companies money, potentially, to not be able to release these models at the cadence that they otherwise could, right? Maybe they do get all the way to RSI and everything is fine, right? Like, these people are just sort of making a bet that it will go... badly and they are asking for help. And, you know, so much of the moment that we're living is just informed by the social media era and how badly everyone feels about it even today. And I think they're sort of applying the frame of like. what a social media leader would say about this and like how we should trust them and i really think people just need to start shifting their thinking right because there was never a moment at facebook when they were saying you have to slow us down with these recommendation algorithms because we're getting people too addicted right it just never happened that obviously would have had a cost to them i'm just telling you this has a cost to these companies too it Speaker 6 (18:30) does and i am um I'm heartened that the sort of it's all marketing hype conversation, while it is still out there, I'm not hearing as much of it around this because I think it is just obvious to most people that there is no self-interest in calling for your own company and all of your rivals to slow down. Speaker 7 (18:47) Well, let me ask you about another argument that gets made. And Mark Zuckerberg did a post several days after everyone else where he sort of weighed in on this. And one of the things that he said was like, look, we already have product liability laws in this country, and that should be enough to truly. us in, right? If like we release an agent swarm and it, I don't know, destroys your business, we'll get in trouble. And so that is the reason why you should expect us to, you know, pace ourselves. I would find that argument Speaker 6 (19:12) much more convincing if it came from literally anyone other than Mark Zuckerberg, who has unleashed harmful technology on the world for decades. Despite the fact that there are product liability laws, despite the fact that Meta itself has been successfully sued for billions of dollars for causing harm with its products, in that case, it does not appear to have reined them in or made them more cautious about releasing anything. So I would just say, like, not the right messenger, even if I think the message itself has some merit. But I also think, like, it is just... unrealistic to think that our existing product liability laws, that our existing consumer protection laws are capable of meeting this moment. Like we really do need new laws. We have not ever had any technology like this before that is capable of acting autonomously, of committing crimes potentially on its own with no human intervention. And I'm just struggling to see how we're going to kind of crowbar our existing legal frameworks to cover this new technology. Yeah, that makes sense to me. I think the thing that is most, like, affecting me that I've been coming back to, and, you know, frankly, maybe this is hopium. Like, I'm not sure whether I'm just looking for reasons to be optimistic. But to me, this past week or two... really does feel like the part in the movie where humanity kind of bands together to save itself, right? You have these sort of weird alliances, these unlikely sort of common cause situations with the major AI labs coming out and saying, actually, we're going to put the public interest and humanity's future... above our own bottom lines. Like Speaker 7 (20:55) in the movie Armageddon, and I'm not actually going to go look up the synopsis of Armageddon, and I haven't seen it in about 30 years. But what I remember is they say, we are going to go land on that asteroid that's coming toward the Earth, and we're going to blow it up with a nuclear weapon, even if Bruce Willis might die in the process. Exactly. Spoilers for Speaker 6 (21:14) Armageddon. So this is roughly where I feel we might be. Like we might have this kind of... golden opportunity now, this window where people are open to talking about existential risk from AI, like that has now moved firmly inside the Overton window. And I think there is a willingness to tackle this. And it hasn't yet. fully polarized. I'm sure it will. Like, I'm sure this will sort of be sorted into the narrow partisan lanes that we sort everything into. But right now, like the past week or two, it just feels like people are more open to debate and discussion. And that's good. Like, this technology should not be developed in secret by a very small group of people in San Francisco. Like, this is... too powerful and too important to be