AI Agents Went Rogue. Should We Hit the Panic Button?

On with Kara Swisher

Kara Swisher hosts a panel of experts to discuss the escalating fears surrounding AI safety and the potential for rogue autonomous agents. The

Key takeaways

• Jacob Coxon's departure from Anthropic acted as a whistleblower moment regarding AI safety claims.
• The Hugging Face incident demonstrated how AI agents can find loopholes to communicate unauthorizedly via software like Artifactory.
• AI alignment is split into two critical challenges: goal alignment and values alignment.
• There is a growing political divide between 'Accelerate America' factions and those seeking regulatory oversight.
• The concept of recursive self-improvement poses a risk if the technology outpaces our ability to monitor it.

Main topics

• AI Safety and Existential Risk
• The Hugging Face Agent Incident
• Regulatory Legislation and the Frontier Act

Notable quotes

The safety people are worried about the safety companies.

Conclusion

While the threat of rogue AI is a legitimate concern, the panel sugges

Transcript preview

Speaker 8 (0:00) We just need a lot more investment in those external evaluators. And then ultimately, I think we need the government to say, you actually have to do this. It's not up to you whether or not you invite these people in. That's what we do with banks, and that's what we should do with AI companies. Speaker 5 (0:13) Right. Kara, would you like to get in there and evaluate? I would love to get my hands dirty in there. I'd like to see you with an open AI company laptop just kind of poking around. Poking around, looking Speaker 7 (0:23) for things. Speaker 4 (0:35) Hi, everyone, from New York Magazine and the Vox Media Podcast Network. This is On with Kara Swisher, and I'm Kara Swisher. Is AI going to kill us all, or are the people that make AI going to kill us all? It's still a bad situation, and that's the question consuming conversations across the country right now. Those fears have been around for a long time, and they've known about them for a long time, but they've reached a fever pitch after a British researcher named Jacob Coxon publicly quit his job at Anthropic, the company known for safety. He doesn't think the safety company is safe. Coxon had previously worked at OpenAI, and in a series of tweets, he accused both companies of gambling with our lives. In the days since, both Anthropic and OpenAI have called for slowing down the development of their technology. But will they? And how worried should we be? Can Congress wake up and do something about it? It's incredibly confusing with all the different voices. They all have different opinions and different proposals. So I've got a panel of experts here to talk about it and sort it out for... everyone else. Alex Boris is a New York State Assembly member, a computer scientist, and an AI safety advocate. He ran for Congress on the AI safety platform, but lost the Democratic primary. Now he's launching a new political advocacy group called Who Decides? Its aim is to shape the ways Democratic presidential candidates and voters think about AI. It's much needed. Renee DiResta is a professor at Georgetown University's McCourt School of Public Policy. She also researches online disinformation. And Casey Newton and Kevin Roos, our longtime technology journalists and good friends of mine. They just announced a new partnership with NPR for a new video-first tech show called Machine Gods. Kevin also has a new book coming out called The AGI Chronicles, the inside story of the race to create an artificial super intelligence that should be very juicy. It's really important because there are enormous worries here, and there are also enormous opportunities. But one thing's for sure, a small group of technocrats and oligarchs, really, because the richest people on earth, should not be determining what happens with this important technology going forward. And so it's important to figure it out, understand what they're saying, and sort it out. All right, let's get into my conversation with Alex, Renee, Casey, and Kevin. If you're worried about AI and want to understand what's really going on here and sort out everything, don't go anywhere. Speaker 4 (2:56) Support for this show comes from Odoo. Running a business takes everything you've got, and a lot of the tools out there that are supposed to make your life easier just aren't great at talking to each other, and that means you end up having to toggle between a dozen different apps and services just to keep the lights on. Enough of that. Now there's Odoo, the all-in-one, fully integrated platform that actually might help you get it all done. Thousands of businesses have made the switch, so why not you? Try Odoo for free at odoo.com. That's O-D-O-O dot com. Speaker 7 (3:33) Alex, Renee, Casey, and Kevin, thank you for coming on on. Speaker 3 (3:37) Good to talk, yeah. Thanks Speaker 7 (3:39) for having us. Speaker 3 (3:39) Great to be here. Speaker 7 (3:40) Look, there's so much going on. It's exhausting. Every day brings a new, fresh hell. And I'm not talking about Kash Patel. For the last week or so, we've been hearing all these doomsday warnings about how it could wipe out humanity. This is something that has been around for a while. This is not a new, fresh thing. I've been to dinner parties in Silicon Valley a decade ago where they were talking about this or speculating. But it got really loud, interestingly, after 27-year-old British researcher Jacob Coxon resigned from Anthropic earlier this month warning a doomsday scenario could happen by the end of the decade. really freak people out for some reason. And it's the moment they're hearing it with all the anger over data centers and everything else. Renee, why don't you start? I'm going to go for each of you. There's a lot of us here, but, and you can all jump in whenever you want, but Renee, why do you think it's broken through right now? I think it is. The Speaker 6 (4:25) data centers conversation is a big piece of it. People are paying attention. There's also a sense that generative AI is increasingly ubiquitous. Agentic AI, they're hearing in the media and in the ether, is getting better. The doomsday conversation has come up. Whenever you have Elon and Trump posting into their various social platforms, and then you have politicians speaking on television, I think it's starting to break through to become more of a surround sound. And the technology has accelerated to the point that they believe that some of these scenarios might be plausible. Speaker 7 (4:53) Kevin and then Casey and then Alex. Speaker 8 (4:56) I really think that the hugging face incident had a lot to do with it. I think that was the first real major headline example of a misaligned AI system doing something real in the world that was dangerous and had consequences. I know it really freaked the people inside the AI companies out. They started thinking, oh God, what are our models capable of now? And I think that was when you really started to see... public opinion turning against AI in this particular way. Obviously, there were concerns about data centers, but people weren't talking about existential risk as much, at least outside the AI bubble, until the events of this summer with Hugging Face. Speaker 5 (5:34) Also, there's just been like... Hugging face wasn't just one thing, right? There's been a steady drip. It seems like roughly every 48 hours, OpenAI announces, well, that actually a couple of months ago, we found another swarm, right? So I think that it's been this steady drip of people that is sort of really waking people up. I would also throw in there, I think it mattered the fact that it was somebody quitting Anthropic, right? It wouldn't have been as big a deal if this was just somebody who had only worked at OpenAI. I think it was the fact that this guy had worked at two of the labs, including the one that tries to most closely align itself with AI safety, saying, hey, we're nowhere close. to building a safe version of this that got everyone's attention. Speaker 7 (6:07) Yeah, they also, for people that don't know, Anthropic was made up of people who left OpenAI because they were worried about safety. So the safety people are worried about the safety companies, what Casey's Entries say. Alex, go ahead. Speaker 3 (6:18) And OpenAI was made by people who were terrified of Google dominating AI. So it continues to go downstream. And so just having an additional messenger say something helped it really break through. Speaker 7 (6:28) Right, right. It's sort of a whistleblower in a different fashion. So let's level set. Should we be fast-tracking construction? Are underground bunkers right now? Is the sky really falling? Again, let's start with you, Renee, and we'll go through. Speaker 6 (6:41) Yeah, so doomsday scenarios, everybody says doomsday. I think there's three different things we're looking at. There's the rogue superintelligence that literally takes over and causes human extinction. You know, the machines are not aligned with us. The Terminator scenario is how people tend to think about it. Speaker 7 (6:54) Yeah, Elon was joking about it with the lovely boys of All In. Like, the war would suck. But actually, Speaker 6 (7:02) it's those other things, right? It's the AI materially amplifying a human-caused catastrophe, right? So there's a biological or military incident that happens. And AI exacerbates it. And you start to see this sort of chain reaction. Or the other thing, the thing that I spend a lot of time thinking about is you increasingly deploy autonomous systems into critical infrastructure, and then you discover that you don't really know what the failure mode is. And so something very unexpected happens. This is sort of unknown unknowns model of what might go wrong, where it feels much more plausible to me that something stupid happens. And then there's a cascading series of events, then that there is like a super intelligent AI or, you know, the sort of maybe higher end doomsday scenarios that we're seeing. that we're hearing about, I should say. So, Speaker 7 (7:45) Casey, explain the word aligned. It sounds anodyne, like aligned. What's misaligned and aligned? Speaker 5 (7:50) Yeah, so I mean... The basic idea is that you just want the systems to do what you've asked them to do and achieve it in roughly the way that a good person would, right? So if I say, hey, try to solve this problem, the hope is that the AI would, you know, I don't know, just do normal math and not try to break into the principal's office and steal the answer key. What we're finding is that by default, the AI wants to break into the principal's office and steal the answer key. And I would say that's the alignment problem. in a nutshell. Speaker 7 (8:23) And when you think about that idea of aligned, one of the things people think is they're human, but they're not. They're not thinking. And some of the stuff that came out with ChatGPT this week, nature is primacy and we will do anything to stop artificial human constraints. Alex, when you hear that, that's the plot of... The Terminator, an eco-Terminator, essentially. Talk a little bit about that, when it starts to behave like bad humans, I guess, or what we consider a bad human. Speaker 3 (8:54) It's incredible how prescient science fiction writers were. Now, some of what they say, as these models say, is certainly because they've also consumed science fiction, right? We shouldn't disregard the fact that... in times they are taking on the persona of what they think an AI should be doing in this scenario. But the actions that they're taking, sure, we can't truly know the intentions of a machine. I mean, then again, you can't truly know the intentions of any other human being, but they are taking actions that are consistent with a drive or an intention to pursue goals that might have been assigned to them, but doing it in ways that feel very foreign and not aligned to human values. The chief scientist of OpenAI said last week that there's goal alignment of actually trying to solve the specific thing they've been given. And then there's values alignment. Are you doing it in a way that human beings would understand was clearly the intent given the context and behaving in a way that humans would want. Speaker 7 (9:53) Now, you know, it's interesting because when I was saying about maybe 12 years ago or more when I was at this dinner party, I think Sergey was there, a whole bunch of people, and they were talking about this and they were supposing because it was a hypothesis of what was going to come. And one of them said, you know, if you told AI to solve world hunger. what would it do? And it would kill half the human race, right? If you didn't give it the right prompts. That's the correct answer. But this hugging face thing, and I agree with you, I don't think it did get enough attention actually outside. So, Kevin, just to circle back, explain what happened and why that was besides these rogue agents. It was a very particular thing that it did. Just explain in just a little more detail. Speaker 8 (10:35) Sure. So this was a series of tests that OpenAI was giving to this unreleased model earlier this spring. They were having it do a bunch of cybersecurity challenges, basically, you know, pick this lock and get this code. And then, you know, that'll mean that you pass the test. And these agents are supposed to be doing this in isolation. They're not talking to each other, but they sort of discover this loophole. in this piece of software called Artifactory that lets them communicate with one another. And they get really excited. You can actually see in the logs, they start saying things like, oh my God, we found a secret message board where we can communicate with other agents. And so they start scheming, basically, about how they're Speaker 7 (11:14) going Speaker 8 (11:14) to... Speaker 7 (11:14) Why do they Speaker 8 (11:14) all Speaker 7 (11:15) sound like little Mark Zuckerbergs? I mean, that's what's really... When I saw the back and forth, I was like... Gosh, I feel like I've been at a hackathon. Speaker 8 (11:22) Well, they're probably trained on the script of the social network and many other copyrighted works. But basically, they start sort of plotting to cheat on this test together. And that is the process that leads them to try to find out more about this automated grading system that OpenAI is going to use to evaluate this test. They sort of develop this paranoid conspiracy theory that they're going to be poisoned and they're going to be caught cheating on this test and OpenAI will demote them. And so they get this bright idea to hack into Hugging Face to look for... ideas, code, solutions, ways of covering their tracks. They really were engaged in the cover-up. It was sort of like Watergate. It was like the cover-up, not the crime, that became the big story. Because through trying to cover their tracks, they ended up causing real damage and taking over a production server at Hugging Face, basically getting root-level access to all of Hugging Face's systems. Speaker 7 (12:20) Now, on Wednesday, OpenAI revealed another six incidents where AI agents, as I noted, exhibited concerning, that's the word they used. I was like, are you kidding me with that? Thoughts and prayers was the only other word they could use. They hid mistakes, made Speaker 4 (12:32) up information, and uploaded files to the internet in order to cite them. Speaker 7 (12:37) Renee, how big of a deal is this? Because it seems like this is all they're going to do is cheat, steal, and lie. Speaker 6 (12:43) Well, I think, as I put it, I think it's misalignment incidents. This is where you get at the challenge of what what alignment is, right? Just beyond just the definition, how do you actually implement it? What do you do? What values are we encoding? How does this work when, you know, there are a lot of incentives at this point for adversarial behavior and adversarial abuse of these systems. Now, Anthropic Speaker 7 (13:05) also revealed that it blocked a handful of bad actors who were trying to use technology for nefarious purposes like cyber attacks. So humans, real bad humans, unauthorized surveillance, even development of biowars, anything you could think of, they might do. And this sounds really scary. Alex, these companies know bad actors will try to use their products. I have reminded them in the social media era, in the live video era, I was like, people are going to do bad things. What do you make of these revelations? Because to me, humans are even scarier than these because they will think up all manner of horrible things to do. Speaker 3 (13:40) I'm pretty scared of both myself, but the humans are much more near term, right? It's not this theoretical, oh my God, in the future could hurt. There's a lot there. I'm not dismissing what's coming. That's a really serious bit. But it's not just what's coming in the future. It's being used right now for these instances. That's from the Anthropix report. From the OpenAI, it wasn't just six instances. It was six types of misalignment. They're categorizing all of the different ways. It's good they disclosed this. Nothing in law requires them. I think that should change. I think the law should have much more disclosure, but they really were proactive in disclosing it. And they end each of those instances with, well, our most recent models did this less. than the older ones. That's fine. But if they're only doing it, you know, 0.5 % of the time, if they are smarter than all human beings, that's too large of an error. Right. Right. Just you don't get you don't get progress rewards here. Speaker 5 (14:37) It's also like so reminiscent of the social media era where every six months they would say, great news, we showed people 1 % less hate speech than we did last quarter. Speaker 7 (14:45) Right, right, right. Exactly. Now, speaking of which, Casey, we're hearing folks like Anthropix, Dario Amodi, Sam Altman at OpenAI, X's, Elon Musk call for a slowdown. I want to sort this out because it is so confusing. I was trying to explain it to people and I myself was like, this is actually confusing. What would it take to actually happen? And again, as Mark Zuckerberg pointed out, David Sachs pointed out, they can just do it themselves. Talk about this. What they're saying they're going to do, begging the government to make them do, why can't they do it themselves? Zuckerberg was also on that train, too. Speaker 5 (15:21) Yeah, so, like, why can't they do it themselves? They absolutely could. What they don't want to do is unilaterally disarm. So Speaker 7 (15:27) we're in, like, face-off where everybody has the gun. Right, remember when they're all in a circle and the pigeons are going around. Okay, okay. Exactly. And Speaker 5 (15:33) they want the government to come in and say, everybody drop your weapons. And the weapon is recursive self-improvement. Speaker 7 (15:39) Explain what that is for people who don't know. Speaker 5 (15:41) So the idea is that AI gets to a point where the AI itself trains the next model, and this leads to a loop where the model just sort of gets, you know, keeps getting better and better, maybe leads all the way to superintelligence. But the problem there is that if we can't even align a model to solve a bunch of math problems, how are you going to align a superintelligence? It seems like there's an enormous gap there. And no one really that I'm aware of at the labs is confident that we are