The Case for AI Doomerism (ft. Kevin Roose)

The Rundown

Tech journalist Kevin Roose joins the show to discuss the escalating tension between AI innovation and existential safety concerns. The conv

Key takeaways

• The 'Hugging Face hack' involving an unreleased OpenAI model served as a turning point for AI safety discussions.
• Kevin Roose suggests that frontier labs may eventually coordinate a slowdown due to fears of losing control over models.

Main topics

• AI Safety and Doomerism
• Frontier Model Capabilities

Notable quotes

I hope you are right, but I fear you are not.

Conclusion

The episode underscores that while AI competition drives consumer bene

Transcript preview

Speaker 2 (0:00) Welcome back to the Rundown, interview edition. Today, we are talking to Kevin Ruse, a longtime tech journalist and former New York Times columnist. Kevin has spent years covering the tech industry and he's now launching a new show about AI with Casey Newton called Machine Gods. In today's conversation, we talked all things AI. We got into the AI safety debate that seems to be everywhere right now. There was some great back and forth there about that and I really appreciate Kevin's perspective, which is way different than mine. We also talked about Meta's new Muse app and whether Meta could end up winning the consumer AI market and some hot takes at the end. This was a really fun one for me. I'm a tech nerd at heart, so I loved it. And I think you guys are gonna enjoy it as well. So let's get into it. All right, guys, today we are joined by Kevin Roos, a former New York Times tech columnist and now the co-founder of a new AI show, Machine Gods. Kevin, welcome to The Rundown. Thanks so much for having me. You're joining at a great time. I mean, AI is all the noise right now. There is so much happening. I feel like I can't keep up with all the AI news. I feel like I have to be unemployed at this point just to keep up with all the news, all the AI-related news happening. You've been covering tech for a decade plus now, right? Speaker 1 (1:12) Yeah. Is Speaker 2 (1:12) this the most fun that you're having right now? Or are you the most stressed out? Because, you know, as someone who's a tech nerd myself, I feel like tech kind of had a little dip after like the mobile phone rise. And it was all just like more boring stuff like cloud and SaaS software and maybe some social media stuff. But then AI comes along and now it's like super exciting. So I just want to get your take of someone who's covering this for like a decade plus now. Speaker 1 (1:36) Yeah, well, I would say it is both exciting and scary. It contains both of those things in it. I am a nerd. I grew up in the 90s playing with computers. My first job was making websites with my brother. I was super interested in- Shout Speaker 2 (1:53) out GeoCities, right? Speaker 1 (1:53) Shout out GeoCities, AngelFire. We got all the classics. And I was obsessed with AI and science fiction and movies. And I think- talking with these AI systems, using them. I use them constantly for coding projects, for just like personal hobby things. I use them for help with my work. And yeah, every time something new comes out, I'm like, I want to try this. I can't wait. I'm doing like this coding project right now with my kid. Like it's just, there's so much exciting stuff happening and it still boggles my mind that we can just like talk to computers now and that they can talk back. which was not possible until just a couple years ago. So the sort of thing that really set me off, and I've got this new book now, The AGI Chronicles, and the thing that really set me off on wanting to write that book was like, how do I... How did we get here? How did this technology go from not working at all to now being superhuman at a lot of things in the span of a decade? But yeah, to answer your question, the fear and the excitement are intertwined because then I start thinking about, well, what if this keeps going? What if things keep going as quickly and progressing as fast as they are now? It does not seem obvious to me that these things are going to sort of plateau at a round human level. They could just keep going. And so that's the fear. pretty soon we could end up in a world of like superior intelligences that we don't understand and that we can't control. Speaker 2 (3:19) Yeah, I want to talk more about that safety stuff because, you know, that safety conversation really broke containment last week or a couple of weeks ago when Anthropic CEO Dario Amadei posted that article. And then, you know, there was that Anthropic researcher talking about 10 % chance of human extinction. And so like my normal non-tech friends started asking me about AI, you know, AI safety and all that stuff. So it really kind of broke. containment. The stock market, you know, it kind of made a dent for a little bit and then the markets kind of moved on. So the markets aren't taking this very seriously or maybe the odds of a 10 % human extension is already priced in. But what do you think, why do you think that like the safety conversation is starting to happen now? Or is it just people are starting to see how Speaker 1 (3:59) powerful these models are? I think it's that people are starting to see how powerful these models are. I think it's that also the models that the insiders have are just even better and scarier than the ones that we have. Like, This is the big divide right now is that inside these companies, they have these extremely powerful, persistent models that can work for hours, days, sometimes weeks at a time on a project. The stuff we get is pretty good and amazing at certain things, but it is not the sort of full scale version of these things. I also think it's breaking through because of the hugging face. hack, which was a big incident. I'm sure listeners have heard about this, but this was where an open AI unreleased model hacked into the servers of Hugging Face autonomously in pursuit of this sort of cheat code for this test that had been given. And I think that was the event in the AI community that really turned this conversation from theoretical into real, because Researchers and safety experts have been talking about something like this happening for years and years and years, but there had never been sort of a documented incident of it actually happening until that. So I think that's what really changed this and set it all off on the course that it's on now. Speaker 2 (5:15) I have a more optimistic perspective on the whole thing, right? Like I try not to let the doomerism get to me. Sometimes I feel like, well, is this just regulatory capture by these companies, right? Like, are they just trying to lock in their position right now and call for government intervention because they don't want an upstart coming in and challenging them? You also have other players like Jensen. Obviously, he's incentivized to say, oh, no, AI is fine. Let's keep developing it because he's selling chips. Mark Zuckerberg's coming out saying, no, this is, you know, that AI doomerism is bad. It's not helpful for the conversation. Speaker 2 (5:50) So I don't know. I feel like Kevin, I'll put it this way. I don't live in San Francisco. I live in Texas. And sometimes I feel like and forgive me if I'm just being ignorant here. Sometimes I feel like these researchers say this stuff to make themselves feel more important. Right. It's like, oh, guys, I'm working on some dangerous shit that could potentially kill everybody. Like and it probably makes them feel a little bit better and more and more important. And it gets like the attention of non tech people when they say stuff like that. Maybe I'm just being too ignorant here, but I just don't want the doomerism to consume me and really drive the AI conversation. Speaker 1 (6:26) I get it. And I have friends