Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

The a16z Show

The episode features a deep debate between industry leaders regarding the future of AI safety, regulation, and security. The participants critique

Key takeaways

• The concept of 'pacing' AI development is criticized as a PR maneuver that lacks clear technical or regulatory definitions.
• AI agent swarms introduce a novel threat model where massive-scale automated agents can perform denial-of-service attacks on internal APIs and services.

Main topics

• AI Safety and Regulation
• Cybersecurity and Agent Swarms

Notable quotes

If you regulate AI too early, you actually don't solve anything.

Conclusion

The conversation emphasizes that the industry must move beyond vague e

Transcript preview

Speaker 3 (0:00) If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately. You willed Speaker 2 (0:07) the Speaker 3 (0:07) thing into Speaker 2 (0:07) being, but Speaker 3 (0:08) you Speaker 2 (0:08) haven't figured out how Speaker 4 (0:08) to control it. Speaker 3 (0:09) The Speaker 2 (0:09) problem we have now is this rift between the labs and the security community that keeps coming to two conclusions. Sloppy, and you're not complete in what you're telling has Speaker 4 (0:20) happened. An employee has like 10 % chance of species extinction. The post is very reasonable. But the atmospherics are Speaker 3 (0:27) not. Agent swarms completely flip that. These are just roaming drones. Speaker 2 (0:32) But like time 10 Speaker 3 (0:33) ,000 Speaker 2 (0:33) and they will easily mistake a good task for a bad one. So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The U.S. about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal Speaker 1 (0:56) that... The early internet was riddled with viruses, worms, and security failures. We didn't stop building it. We learned how to make it safer. What should AI take from that history? In this episode, I sit down with Aaron Levy, Stephen Sanofsky, and Martim Casado to debate AI safety, regulation, and what changes when agents start operating across the software we use every day. We get into why agents could force a rethink of permissions and cybersecurity, what decades of software security can teach today's AI labs, and why regulating a technology before we understand how it actually fails can create problems of its own. And we look at a broader shift already underway. As models mature, some of the most important AI innovation may increasingly happen outside the frontier labs, in the systems and software built around them. Speaker 1 (1:49) Guys, welcome back to the podcast. Thank you. Didn't think we'd ever do this again. Speaker 2 (1:54) I can't believe it. This is great. Speaker 3 (1:56) I mean, Martine's just building these $100 billion companies too busy for this podcast. Or at Speaker 4 (2:01) least taking credit for it, Speaker 1 (2:02) as Speaker 4 (2:03) VCs Speaker 1 (2:03) do. Exactly. We have a lot to discuss today, but Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed? Oh boy, Speaker 3 (2:13) I think we should start with Martina on this one. You were fighting lots of good ground wars. Maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. I think we would agree that any AI lab right now at the frontier should be building in the safest way possible. with the highest degree of governance and security and whatever your definition of alignment is. This is an incredibly important area of research. It's Speaker 1 (2:38) an Speaker 3 (2:38) incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high quality products that can be trusted by enterprises and that aren't kind of constantly hacking systems. So when at least I read the Dario post, I actually didn't disagree with almost anything because it was all about how do you have better security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers and do you agree with who those are and does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is, does this get sort of used or leveraged? to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it easy to compete with the frontier labs. Or do politicians kind of end up sort of taking the message and run with it and maybe even worse outcomes happen? It's used to ban data centers far faster and whatnot. And so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general. And then the question is, what do you do about it? especially what do you do about it from a regulatory standpoint. And that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on let's make sure that we don't use this for regulatory kind of capture, which I also agree with. But I think the ideas in the pacing conversation are important. Again, it's a little bit of a funny concept because maybe it's not even pacing as much as just good hygiene and engineering. And so with good engineering, obviously there is a slight slowdown. But it's Speaker 4 (4:08) a Speaker 3 (4:08) slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would trust using it. The Speaker 4 (4:16) post is very reasonable, but the atmospherics are not, right? So an employee is like, this is going to kill, whatever, 10 % chance of species extinction. And you know what Dario says? I agree with him more than I disagree with him, right? On Speaker 2 (4:32) TV. On TV, the same Speaker 4 (4:34) day that he landed these things. And so in some way, you can't have these conversations in isolation, which is, of course, if he's going to agree in species extinction, this post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. And then I have this quibble. but it really bothers me because I'm a pedant, which is, I think pacing is the wrong way to describe this. For one, it is orthogonal to security, Speaker 2 (5:07) right? Speaker 4 (5:08) So you can very slowly build a nuclear weapon, and that doesn't make anybody feel better than a slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually... addressing it. So you're saying, well, we're not going to pause, we're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy. Speaker 2 (5:30) Because Speaker 4 (5:30) the pause people are like, well, that's not a pause, that's just pacing. And the regulators are, you're still doing the thing. Speaker 3 (5:36) And Speaker 4 (5:36) so I just feel like my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction, which are doomers, and then the regulators on the other side. And the problems are making both of them unhappy. And I think what they have to do is they need to come out. They need to address the X risk question directly. They need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes very sensible. And Speaker 3 (6:10) what would you do just to play the other side for one second? What do you do? Do you make room for the one possible Venn diagram, which is the lab researcher that is both simultaneously super scared, but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that. And then obviously the language is right now very problematic, but that person does exist. And that is a real kind of person in our industry, which is like, we have to be at the forefront of AI. I'm also very scared of it. And so that's why I'm working on this. Speaker 4 (6:45) So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have existential Speaker 3 (6:52) risk. Speaker 4 (6:52) You were the first Speaker 3 (6:53) pacer. Speaker 4 (6:55) We were part of the first pacer. So if a constituency within the labs that are the most knowledgeable people believe this stuff has existential risk, the answer is to nationalize it and actually put controls that we know that work. Right. Now, if they don't actually believe that and in the private conversations I have, the most sensible people don't. It's Speaker 3 (7:17) a Speaker 4 (7:17) small fraction that do. This is an HR problem. Right. So to me, an HR problem is a company problem. Like if they are worried that they can't recruit people. Right. Or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case. I think that is the wrong reason to cause a national level lockdown on a very promising technology. So listen, I think the post, again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on X risk with the proposal that was put out. You just can't reconcile those two things. And that has always been my primary. Right, right. Okay, Unleashed. Speaker 1 (8:03) He's just holding it back. Speaker 4 (8:05) So, Speaker 2 (8:06) okay, the first thing is, is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen, they haven't. So you can't pace it because nobody knows when Speaker 4 (8:17) it was supposed to finish Speaker 2 (8:18) in the Speaker 4 (8:18) first place. It also just seems disingenuous. It's complete nonsense. 100%. Like Speaker 1 (8:22) you Speaker 4 (8:22) can't Speaker 2 (8:22) claim, this is like when the press reports on Apple's latest iPhone is late. From what? Speaker 3 (8:28) Nobody Speaker 2 (8:28) knows exists. They haven't told anybody. Yeah, my Apple car was very late. Yeah, like, I don't understand. In order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense. And you can't escape that. And then... Speaker 4 (8:44) By the way, this is another problem that, like, again, from my little quibble on PR is nobody believes it anyway, right? And so that's the thing... Wait, which part? The pacing. That they're going Speaker 1 (8:56) to pace? That Speaker 4 (8:57) they're going to pace. I see, okay. They've been at a dead run. They've raised more money than ever before. They've run faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your... Speaker 3 (9:05) I see, I see, yeah. Well, the issue, obviously, is that the model that everybody has internally far exceeds what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point. Pacing versus what? Right. Speaker 2 (9:16) We don't know if the one that scares everybody also doesn't work for a legal brief. It might actually screw all that stuff up because it's so, who knows? Right. So there's that. And it's just disingenuous to claim that you're paid. Second, why do they have to announce all of this and ask the government to tell them to pay? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product. Just stop. Don't do it. Like, I worked at a missile factory in college, and we had nuclear missiles, and I walked the floor. What's Speaker 4 (9:47) with Speaker 2 (9:47) you guys and missiles? I'm Speaker 3 (9:48) just here with software. You Speaker 2 (9:51) were one of two people in college when we were, which was either you were protesting to keep them off campus. Or building them. Or building them. So that was your choice. Like, look at that. Speaker 2 (10:06) nuns from the Catholic Church show up and pour blood all over on missiles. Oh, totally. You know, and I'm busy just wheeling PCs around on carts saying, here's Speaker 4 (10:13) a Speaker 2 (10:13) secure PC. And I'm like scared to die. I didn't know what was going on. And I'm like, it's a nuclear missile. And then you find out that that's what stopped the Cold War. Like that literally, it was a Persian missile. And that was what did it. And so, but you said something that I think is super interesting, which is pacing is this sort of... fuzzy non-word between like going and not going. And the problem is, you're exactly right. Like, there's no one who's going to be happy with the middle road. And so one of the things that people, I think it's almost fun for me to watch as a sport, they think that all these people saying to the government, do this, don't do this, that they think they're going to get what they want. And it's a complete, 100 % misunderstanding of how government works, which is you, When you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that all the inputs can make it into the government, the output never makes everyone happy. Right. 100 % of the time. It either doesn't go far enough or it goes way, way too far. Yeah. And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like, it was, was it David Sachs Speaker 4 (11:42) was like, I think it was David Sachs, but someone from the government said, you're asking us to regulate you. No. So basically the answer was no. Speaker 2 (11:54) That's Speaker 3 (11:55) probably just Trump, I think. But the Speaker 2 (11:56) thing that they know now, that you just know from experience, is once the wheels start on regulating. You can't slow Speaker 3 (12:02) that one down. And Speaker 2 (12:03) now it's become an election issue for every party in every jurisdiction, up and down the whole government stack. So there's now this whole. basket goods, a basket of regulatory approaches. Well, Speaker 3 (12:14) the next election will 100 % be a referendum on AI. So, like, it has to happen that 2028 is, like, the AI election. And you can basically run on... The problem is, like, it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it. or at least try and like avoid the topic. Yeah. But it is too bad that we as a country are in a spot where like the pro-AI case is just like it sounds too, it's just like takes too many words. You know, it's way too nuanced. Speaker 2 (12:52) It's Speaker 3 (12:52) defensive. Speaker 2 (12:52) It's Speaker 3 (12:53) defensive. And Speaker 2 (12:54) we own none of the vocabulary. Right. So the whole debate is pause, it's swarms, it's rogue. Every word has been chosen by the people who don't want to do AI. Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that... Yeah, so we got to be like Speaker 3 (13:15) union jobs and, you know, cancer and like, you know, we need, you know, there needs to be another word cloud that emerges. What Speaker 4 (13:22) I don't understand is why the labs have not taken a position on X-risk. Like, short of that, I don't think this goes in any direction other than heavy-handed regulation. Yeah. Well, it would just be negligent of the government to be like, there's a 10 % chance of species extinction. The CEO of the top company says he agrees with that. Like, how can a government not have had a regulation? Well, but what Speaker 3 (13:44) would anybody really, knowing this ecosystem, though, I don't know that you would be able to pin anybody down on that other than something. Speaker 2 (13:53) No, Speaker 3 (13:53) I Speaker 2 (13:53) was saying Dario said it on. No, but I'm saying you're not going to pin anybody down on a lower number. Well, in fact, he. does the worst thing about it, which is he agrees with people who claim that they believe that there's an X percentage of extinction happening, but he specifically is out of