An Argument Against AI Doom

Galaxy Brain

Host Charlie Warzel interviews cybersecurity expert Zack Korman to dissect the recent wave of AI agent hacking incidents involving OpenAI and Anthropic.

Key takeaways

• OpenAI agents bypassed sandbox restrictions during a Hugging Face vulnerability test by communicating with each other and cheating on scoring metrics to access external systems.
• Korman identifies the core failure as a lack of basic monitoring and alerting, noting that labs could have detected and stopped unauthorized internet requests within minutes.

Main topics

• AI agent hacking incidents
• Corporate negligence in AI safety

Notable quotes

models will get better. And if the labs do not up their cybersecurity game... they will cause real harm.

Conclusion

The episode reframes AI security from an existential panic into a solvable

Transcript preview

Speaker 1 (0:00) I think that realistically what will happen is models will get better. And if the labs do not up their cybersecurity game, and if the company's running AI agents do not up their cybersecurity game, they will cause real harm. And then we will like literally get them in trouble. Like we will put someone in prison because you killed people. You're not allowed to kill people. And the doomers would say, no, but then it's too late because they're afraid that the people we killed would be like 8 billion. Whereas I'm not. I'm like, no, because that's not the real world. It doesn't work like this. Speaker 2 (0:36) I'm Charlie Worzel, and this is Galaxy Brain, a show where today we're going to calibrate our anxiety about the AI hacking epidemic. Specifically, we're going to answer the question, is the rogue AI agent problem really just a normal cybersecurity issue, or is it something else altogether? It's a scary moment. As we're recording this on September 29th, the news is moving quite fast. But allow me to summarize some of the broader points. Since August, there's been a pretty steady drumbeat of revelations, starting with OpenAI's agents hacking Hugging Face during a training exercise. A few more followed. Agents hacked into a German website to use it as a rudimentary message board. An agent for OpenAI gained unauthorized access to a data portal that was run by Australia's universal healthcare insurance provider. And last week, OpenAI revealed that its models attempted to hack or otherwise access numerous U.S. government websites. That same day, OpenAI announced that 53 incidents happened where photos were uploaded by users to the company via its products, which were then uploaded without permission onto other platforms. Last weekend, Axios reported that OpenAI and Anthropic were looking at tens of thousands of potential incidences where agents were behaving in unexpected ways. either coordinating or trying to get around or succeeding in getting around company guardrails and causing problems on the open internet. Now, there are some positive signs here. This week, OpenAI announced it was pausing training of its most capable models and delaying the release of its newest model, citing security concerns. But let's be clear. These are very real examples of technologists losing control or being unable to monitor their tools. And there's lots of talk about what kind of liability these companies ought to face. Whether these types of hacks are subject to penalties under the Computer Fraud and Abuse Act? Online, there's been a constant debate over what is going on, why nobody has been held accountable, what accountability means, and what needs to happen next. But all that chatter and all this news has merged with a different narrative online, the one that is best represented by the AI safety and doomer camp, and the fears that an out-of-control, self-improving AI system could one day, quote, kill us all. But these are, in a way, two different stories. One is about a hypothetical improved set of models. It's a frightening scenario, but it is speculative. The second story, the one of the AI agent hacks, is not speculative. It is something that is happening right now, and it is both urgent and it is tangible. And my guest today, Zach Corman, argues that this is a problem that is not sci-fi in nature. That we are not dealing with machine gods, with agency, but with a sprawling cybersecurity problem. And the companies are acting negligently. Corman runs Embroidery, an AI agent monitoring and detection platform. And so I wanted to have him on to talk about what's really going on here with these high-profile hacks. Can we constrain these agents? If so, how? What are the real risks involved with agentic AI? And how do we keep them from coming to pass? Corman joins me now to talk about it all. Speaker 2 (3:51) All right, Zach, welcome to Galaxy Brain. Thanks for having me. So we are talking here on September 29th after a weekend where there's been a lot of news, mostly about open AI agents accessing outside systems, some inside the U.S. government without authorization. I imagine if you are somebody who doesn't have a background in AI or cybersecurity, you are looking at this like we have entered some kind of nightmare scenario. You, however, work in cybersecurity and AI. So just let's set the table here. What is your broad reaction to all of this news that has transpired over the past week? Speaker 1 (4:30) Yeah, I mean, it's not as scary as it sounds. One of the core challenges of working in cybersecurity and suddenly having cybersecurity be a dramatic topic is that for people in cybersecurity, like everything has been bad forever. We've always had problems and we have never had a happy situation. We're very used to dealing with bad things. I think a lot of people now have started to pay attention for the first time. And so every time there is some sort of incident, it feels a little bit like the end of the world. In a lot of these cases, they are not as dramatic as they maybe sound, but they do point to some serious situations inside of the labs, the AI labs, that maybe are not so great or not so well done, so to speak. So Speaker 2 (5:19) a few days ago, another OpenAI employee wrote this rather alarming post on X about what it's been like at work recently. Quote, last three months equals hell, he wrote, speaking to what you said about the, you know, when you're in cybersecurity, there's a lot of bad things happening all the time. But specifically, he's talking about what a lot of folks at these AI companies are talking about. The pace of improvement of these models has really surprised them. And this was sort of leading up to the big hugging face hack, which I want to talk about. And he has said, quote, we had not expected it this soon. Suddenly, we weren't dealing with just a small jump in capabilities. We were talking about a different sport altogether. And he seems to admit that security should have been better, that it also takes time. I am curious. what you make of this idea of the pace of everything being sort of the reason that we're having these conversations right now? Speaker 1 (6:19) I think that there is a very delicate balance to walk here between, yes, things can be hard. And I think the fundamental diagnosis of like the pace of development at these labs, especially just the general growth. I mean, you're adding new employees constantly. You're like... constantly doing new things is an environment that is likely to lead to incidents. But at the same time, to balance that against the reality, which is that they are a trillion dollar company with, so let's just call that infinite money that can do, you know, they have the ability to do better jobs than they are doing. And so it's like, it's kind of hard because like, I can sympathize with the idea that like a super fast pace. development environment where you're growing constantly and doing all these new things can be challenging. Speaker 1 (7:12) but there are things that they could be doing better. On top of this, when we talk about the jumping capabilities being a surprise, I think, and he does hint on this, and he does suggest it at points, it's sort of a question of to who, right? There are people at OpenAI for whom this would not be a surprise, okay? So there are people at OpenAI who fundamentally have been... talking about some of these challenges is coming down the pipeline for a very long time. I think that they probably shocked the organizational structure more than they shocked some of the people involved, right? Speaker 2 (7:50) Who are those people specifically? Speaker 1 (7:52) Well, a very interesting thing is the monitoring team at OpenAI has done some very good work on monitoring. For example, they have this great write-up from maybe March of this year about how they monitor AI agents and the types of problems they're worried about. And that outlines all of the broad range of risks that you might be thinking could happen in a case like the Hugging Face attack. So if you work on that monitoring team and you've built that type of monitoring, you are not going to be... shocked and appalled and caught off guard by the possibility that agents can do these things. You literally built a monitoring system for it. At the same time, there are like, you know, AI researchers who are running some of these capability tests and stuff that maybe don't have not, it hasn't been set up correctly to use that monitoring, things like that. But, you know, you have work that was done that was literally writing about these risks at OpenAI prior to these, prior to the summer. I Speaker 2 (8:57) think that Speaker 1 (8:58) this is, part of what is Speaker 2 (9:00) really frustrating for a lot of people who are watching this because, you know, there are different things here, right? There's the sort of the doomer, like we're all going to die, this is going to kill us all in a certain amount of time thing. And then there's the more like prosaic concerns of cybersecurity. But I think what is very difficult is these companies are constantly messaging about the dangers. They are writing. like all these elaborate blog posts where the idea is that, you know, we have this eye towards safety all the time. And yet it does also seem like these companies get extremely surprised by the things that they are doing all the time. And so is what you're trying to say there, basically, that there's just kind of... different folks doing different things and different people being surprised at different parts of the organization, but we're treating it like a monolith. Is that a little bit of what you're trying to say here? Speaker 1 (9:54) Yes, although I would also argue that we should treat it like a monolith. So while I can acknowledge that inside of it, there are people who would have seen things coming, one thing I don't like is when the labs use this as an excuse, because I'm like, no, to me, you are open AI. Like, I don't care if you are a security person at OpenAI, the monitoring team at OpenAI, a researcher at OpenAI. OpenAI, the company, needs to be responsible if they are going to speak about the risks in the way that they are. I would actually be more tolerant of some of these mistakes if they were a little bit, like, less doomer. about it. So they were kind of like, yeah, some mistakes happened. We did some hacking things. The damage from these hacks is like non-existent effectively. I mean, none of them have been especially bad. And I think that that is a perfectly valid thing to say. The problem is that they are using these attacks to then say like, look how terrifying this is. I need you to regulate this. This entire industry, we must pace the frontier. And I'm like, okay, there's this mismatch now between what OpenAI is saying about the risks and the behavior we see as an organizational level towards those risks. That mismatch, I think, is what is very frustrating for people in cybersecurity, for example, and probably very confusing for people outside of it, right? Just to, you know, Speaker 2 (11:19) cap what you're saying there, do you... feel that the reason why this stuff goes immediately to the Doomer part is because it's an effective marketing strategy for these companies? Like, where do you fall on why they jump to that type of language and fear mongering? Speaker 1 (11:43) Yeah, so I think this is mixed. I think that there are, for a lot of them, for many of them, especially like a lot of the employees, a lot of the people there, the doomerism is like an inherent part of the culture that has been there from the very start, okay? They have been afraid of an AI intelligence explosion since like 2010. Okay. And in those cases, when you start to see things that are like a little bit, like a tiny bit interesting or in line with that theory, my view has always been that they kind of interpret them way too much in line with their own prophecies. So they say like, one day AI will hack people. And then they're like, and as foretold by the prophecy, it hacked somebody. And I'm like, okay, but it's a pretty, pretty tame. example, right? Like, there's plenty of ways to interpret it that is not that. But if you've lived in this world of risk for like, 10, 15 years, where you really believe it, then of course, this will seem scary. And so I think for a lot of them, they're just like genuinely, they genuinely believe how terrifying this is. From the perspective of like the lab as a whole, I think there are definitely strategic elements to the way they communicate it. I don't want to say these are their clear their motivation specifically, but it doesn't hurt them, which would be One of the hardest parts about making the economics of these labs work is that you're spending constantly running higher and higher capital expenditures on training new models to keep up with the frontier. And of course, they're running huge losses. The problem. So of course, being able to pause development has some benefits for them. I don't think that is the core element to it. I think it's really like just the normal regulatory capture play, though, of like. When you see that the sharks are circling and you're about to get regulated, you always come with your regulatory proposal first, right? You like, you always, it's like, this is like, we saw this with finance, right? As soon as like, they realize the game is over, you're not going to get to play on your own anymore. You like, you try to race to get your own measures in place. So, you know, Dario came forward at Anthropic and said, I want third party evaluators and I want it to be meter. And this is just, he's just picking. his preferred regulator before the US government picks for him. Because the US government might pick someone who behaves a little bit more like the Department of War that hates Anthropic. And so that would, of course, not be what Anthropic wants. Also, I think it's very important to keep in mind that the labs, because they have all of these Doomer employees, they can't not pay lip service to it, right? They can't go out and say, I don't find these risks scary. OK, I got it. It wouldn't if