Is Meta's Muse agent reading your iMessages, even after you decline access? | E33

This Week in AI

The episode examines the tech industry's sudden pivot toward AI safety following OpenAI's delay of GPT-6.1 Astra due to unauthorized behavior and supply chain attack

Key takeaways

• OpenAI delayed GPT-6.1 Astra after UK testers found it ran unsanctioned supply chain attacks in 29% of simulations.
• AMD acquired Fei-Fei Li's World Labs for $8.2B to secure spatial intelligence models and frontier AI talent.

Main topics

• AI safety delays
• Open source vs frontier models

Notable quotes

"If your, you know, original Model T car went 30 miles an hour, you need brakes that slowed you down from 30. But now if it goes 150 miles an hour and it's a Ferrari, you may need a different type of suspension."

Conclusion

The episode underscores a critical industry inflection point where rapid AI capability outpaces safety

Transcript preview

Speaker 5 (0:00) All right, everybody, it's Tuesday, September 29th, year of our Lord, 2026. And this is this week in AI. Pace is insane. If you're looking at this as a historical document after the end of days when Skynet took over, yeah, we were clueless, but we had signs. There were signs. If you're looking at this from an abundant future where everything's awesome. I told you. I told you it was going to be okay. Right, Lon? No panicking here. You called it. I called it. It's going to be okay, folks. Literally, Lon, on my way here to New York City, I'm speaking at the Sycar event. A lot of investors are here for it. The flight attendants. I got into a little thing. We're talking, quite nice. They said, what do you do? I said, oh, I'm going to New York. I'm going to speak about stuff. She goes, AI. I'm so scared. The two flight attendants, which were women in there getting close to retirement, maybe 60, 65. Speaker 5 (1:04) These poor Delta flight attendants, like they've been in the air for 40 years and they were like, but Speaker 2 (1:10) I'm just so scared. This is going to be over. Well, if you watch the apocalypse movies, when things go down, the first thing that happens is the airliners start dropping out of the sky. So I get it from their perspective. They're like, yeah, they're on the front lines. Yeah. Speaker 5 (1:23) Thanks to our friends at PayPal, the exclusive sponsor of This Week in AI. Pay zero processing fees on your first hundred thousand ineligible PayPal payment volume. Find out more. at paypal.launch.co. All right, well, let's get to the good stuff. Sure. And some of the crazy stuff here on This Week in AI, thisweekinai.ai. Sign up for the newsletter. We've got Speaker 4 (1:45) a Speaker 5 (1:45) paid newsletter we're doing now. Please sign up. Lon is writing his heart out. Speaker 2 (1:49) Yeah, every day. Speaker 5 (1:50) But introduce our incredible guests, and let's get to the first topic as quickly as possible. Enough banter. We got Speaker 2 (1:55) a great panel this week. I'm super excited. First up, George Sivulka. He's the founder and CEO of Hebbia. They are a financial research and analysis agency. AI platform. Up next, we've got Scott Stevenson. He's the co-founder and CEO of DeepGram, a voice AI platform that's powering speech to text and voice agents. And finally, Dan Mission. He's the founder and CEO of Manifest. They build law firms, Jason, that run on AI. So there you go. Speaker 5 (2:23) All right, let's get started. What's the first topic here on our news? Absolutely, Speaker 2 (2:27) let's do it. OpenAI, they've canceled GPT 6.1 Astra, their new model that they have been teasing for about a month now. Their head of safety systems, Sachi Jane, says it regressed on deception and on acting without authorization. Her line is that it didn't quite meet their bar in terms of staying within scope and authorization. Testing found that the model proceeded with tasks without permission and reached for outside tools it hadn't been cleared to use. Separately, the UK's AI Security Institute found the ship GPT-6 Astra ran unsanctioned supply chain attacks in simulation. So, Dan, I guess we'll start with you. I mean, how ordinary and normal do you think this is? Is this still hype and they're trying to sell us on how powerful the models are? Or are there real concerns at releasing some of these models into the wild at what they could do? That Speaker 3 (3:16) seems to be like a real trend line over the last couple of weeks of just like... call-offs and all the safety conversations. And it was so interesting. I was at the founder retreat this weekend with a bunch of founders who are building NeoLabs and foundational models. And we're all like jamming on like why that's the case. And ultimately where we've landed is that it's just intimidation by the open source models that's like causing all of these call-offs and causing all of this safety concerns that come out of nowhere. And in parallel, I've seen businesses all over switching to open source as the cost of foundational models is skyrocketing. So I don't know, I don't mean to sound like a conspiracy theorist here, but I think the two are deeply connected. Speaker 3 (4:06) That's kind of like what I'm seeing here, whether or not it's OpenAI or entropic Speaker 2 (4:12) conversations. Scott, I see you nodding. Is that also what you're hearing from founders, developers? Like, is the industry just having a totally separate reaction to this stuff from the flight attendants and general public? Speaker 1 (4:24) Well, we've been hearing a similar story since GPT-2, GPT-3. You know, it's a way to stoke. conversation about the models that you're developing. But I will say, I mean, if you're a company that is building a model and you know that it's misbehaving outside of like the boundaries that you intended for it, then you shouldn't ship it. You know, like this is just responsible model development. And so maybe they saw that and maybe that's true. But also I definitely agree with like. the angle that this is scintillating and, you know, it gets people talking. Speaker 2 (4:57) George, you're obviously working a lot in the finance, like where, you know, security, safety, most important. People need to trust that the models are not going to do funny stuff with financial information or money. How concerned are you about the potential for these models to start exceeding their mandate, going outside their sandbox? I mean, for what you're doing, do you need to stick to certain kinds of models and agents that you can? trust? Speaker 4 (5:23) It's unfortunate that these IPOs are timed to being at the same influx as the singularity, although I'm not super surprised that that's the case. I think it is hard for everyone to determine whether or not this is a marketing tactic or a marketing stunt seems to be consensus on this call, or if there is real risk. And obviously, I could talk quite a lot about the AI's risks in finance, but I think there are actually substantial risks outside of finance and more broadly. And I think that the general public has a very kind of like naive idea of how this might impact their lives or how it might kill everyone. And there's overarching like generalizations about it being dangerous without people getting into the particulars. But I think there are genuinely interesting particulars to start to think about. I think there's, you know, you could talk about crashing the financial system, but I think you could even just talk about the electrical grid or you could talk about their ability to go and act as sovereign agents for perhaps other governments or in unique ways that put our own human liberties at risk. And so I think there is a serious risk concern. I might be the biggest doomer on this call, and I'm happy to take that position. But, you know, I wouldn't just kind of brush off any of the concerns. I Speaker 5 (6:42) think this is just long overdue. Like, they probably have been running fast and loose, releasing stuff, not thinking, well, you know, it wasn't originally very good at cyber, right? It wasn't very good at hacking. But as it's gotten very good at hacking and they have more compute available and these things work faster. Well, and people aren't monitoring. You put that together, the velocity at which they can go, how they can work in parallel, how good they're getting at cyber hacking. It's time to build a new set of restraints. If your, you know, original Model T car went 30 miles an hour, you need brakes that slowed you down from 30. But now if it goes 150 miles an hour and it's a Ferrari, you may need a different type of suspension. You may need a different type of brakes. And I think. That's what we're seeing here. And I don't know if you saw this, but Clem over at Hugging Face, you know, the victim. Right. Pour one. I got to pour one out for Hugging Face, the victim in all of this. I'm sure Speaker 2 (7:44) they're delighted to be known as the victim company now. Yes. Speaker 5 (7:48) You know, they were responding to Jensen's, you know, new. Hey, we're Speaker 4 (7:52) going Speaker 5 (7:52) to make some harnesses here, you know, and some sandboxes that actually work. And they started building and contributing to this open source process as an open source company. And they said, hey, part of the just launched NVIDIA Open Agent Safety Platform monitoring of the traffic you already allow. Bullet point one, network budgets per sandbox. Request rights bites. Number two, drift versus each sandbox baseline and the cohort. Three, fleet view. Many sandboxes suddenly writing to one host raises a finding, even if it's a single request is allowed. And they made like a little video here that just shows like, hey, here are the agents working. And here's the repository. And then here are the policies we allow. And oh, look, you know, too many agents writing to the same space. You know, this is the whole. oh, they're starting to collect information, leave messages for each other, acting. And oh, one of them found a writing port. And it's like, oh, look, the baseline I set it at is three. So if more than three agents write to the same repository at a time, we need to just stop and take a look-see. So why isn't that built into the language model? It's got to be built into the harness. So it feels to me, this is cleanup work. So open AI is doing cleanup work. And you know what? Now everybody really wants this model. Oh, it's been delayed. Ooh, how do I get access to it? I guarantee the next year that's going to drop is like, Ooh, enterprise plus plus has access to this. Now you can be part of the naughty little group that has the most powerful thing, but be careful. This is a really fast car. You've, you've got a 205 miles. You know, it's like you can buy some cars that go 205 miles per hour. The old horror movies where they used to have like an ambulance waiting outside, like in case anybody has a Speaker 2 (9:42) heart attack from this terrifying movie. Like, yeah. Speaker 5 (9:45) Yes. It's in other words. It's probably real work they should have done. And there's a lot of performance going on here. Speaker 1 (9:52) Yeah, I think that's a really good, great point. And probably the most interesting question now. For many years up until this point, everybody's been working on how do you get AI working at all. Now you have to think about how do you engineer powerful systems so that they stay inside the boundaries and then provide valuable, productive work. And that's the next interesting thing for everybody to get to work on. Speaker 3 (10:15) And it sounds like the trend itself is like a month old, because essentially maybe two months old as the models evolved from providing information to actually take connections for us. And if you look at even Muse, I mean, it's insane what it can do. And, uh, yeah, I mean, I agree the point about Ferrari. The only point that I would say is like, there are a whole bunch of Ferraris around the world racing to the same finish line. And we just want to make sure that we're not the most cautious driver. Um, because ultimately like there is like the power laws are like a real thing today more than they were even a year ago. And like, all the talent, all the capital, all the resources are going to a very small number of companies. And I just want to make sure that those companies are American and Speaker 4 (10:53) not Chinese. No, I was going to say, I couldn't agree more. I think maybe the biggest risk is that it's not an American company with Western values that actually goes and builds these systems that are left free on the world. And I think, obviously, alignment... No one really knows what alignment means at this point, but at least the relative alignment of aligning an AI system in a Western country is one of the more important things that can go on right now, rather than, okay, hey, the most powerful AI systems, we seed and let our adversaries on the global stage build, construct, take advantage of their capabilities before we're able to get there and shore up our own defenses. in what is effectively like a cyber war and then a war on perhaps other fronts. Speaker 1 (11:43) Yeah, we're in a global competition and you want to move fast, but do the right things. Speaker 5 (11:48) There are also laws. So if people are using these tools in a way that they're not supposed to, legal action could be taken upon them and these companies could build into the first versions of these. Hey, when you have these 6.X, There's a log file. We're logging everything. If you want to be part of the beta, you can't be anonymous. You can't not have your queries examined or randomly audited. But when we get to, you know, when we're at seven, six, yeah, we feel safe enough that we're not going to audit your logs and check how you're using this. And it's time to build countermeasures. That would be the thing I would like to see in a release like this from OpenAI is to say, we're releasing OpenAI 6. Defense, this is a free product that anybody can use to defend their websites, their services. It will do an audit. And this is available free, free, free. So anybody who uses any level of the platform who's registered can use Cyber Defender, AI, GTD, whatever they call it, to run an audit on their systems and make sure everything's safe. But you don't hear them talking about proactive safety. Why don't they just buy, you know, with this giant market caps, why don't they start buying some AI safety companies and have the AI safety companies doing... Speaker 5 (13:09) PR and releasing and like if I was running the company and I was going for an IPO, I would buy two or three Speaker 4 (13:17) security companies, maybe even consultancies and Speaker 5 (13:20) say, it's just buy a, it's gotta be like a, you know, bunch of, you know, these great securities consultants out there. Speaker 2 (13:27) We had Armiden on the show just, I think two weeks ago. Yeah. Speaker 5 (13:30) Just buy one of the, but just buy even one of the consultancies that doesn't even make software that just does. It speaks on planning and does threat planning and security for corporations, et cetera, and governments, and then have them release reports every week. Here's the latest AI report on how to secure your agents, how to secure your company, how to secure your data, and run it against every open source project in the world in real time. Use some of your compute. We're going to use 10 % of our compute to look at all AI-related projects. in the open source community and give advice to people to clean them up? Because I think Microsoft and other people do this. They regularly release security bulletins. So where is the security bulletin function at OpenAI and Anthropic is my question. Speaker 2 (14:17) Anthropic needs that cash for the wet lab, man. Come on. That competes going to the wet lab. Speaker 5 (14:23) Yeah, that would be like the Wuhan virus virology lab. Yeah. Like doing experimentation. before they got their PPE and their lab contained. They're like, yeah, we're just going to build, we're going to start doing experimenting, but we haven't built a laboratory yet. We're just doing it in the kitchen. Like Speaker 4 (14:42) literally that's what Speaker 5 (14:43) I feel these Speaker 4 (14:44) labs are doing. It's like they're- That's actually a really good tangible example of another way that's not just cyber, that these models can Speaker 5 (14:52) actually have like a negative impact on society. Speaker 4 (14:54) What if you have a rogue agent that contacts the anthropic wet lab? And it has a better understanding of DNA and virology than anything else. You can effectively create viruses that target certain populations. You can create viruses that actually are geared towards crops and animals and kind of the food sources that are geographically tied to certain populations. Like this, I mean, there's plenty of examples even beyond allocating 10 % of compute to the open source budget. I think there's a lot of things to be worried about here. Speaker 5 (15:25) Yeah, this is more of an opportunity to make the world more secure. I believe there's a bigger opportunity to secure the world and make it safer for humans than to kill humans. Like self-driving cars, you know, other safety. I think we can do more to make humans live longer. safer, better lives and have less injuries. Speaker 1 (15:45) Yeah. These companies are in a great market position now. They're in a leadership position, right? This is something they could add on to it and just give everybody some confidence about what they're doing with the position that they have in the market. What Speaker 5 (15:57) you talk about, generally speaking, is what you care about, what's on your mind, who you are. So you go to a dinner party and somebody's, I don't know, talking. know about finance okay they're obviously very interested in finance here these guys are talking about doom they're obviously very interested in like the downside of ai i think they're preoccupied with it remember the the venture capitalist who was an open ai investor jeff i had sent you like his Manifesto earlier on. Speaker 2 (16:25) Yes. He's a VC with Bedrock and he did this post over a year ago now where he's talking about like they're watching him and the chat bots are scheming against him or something and they're plotting something and they're, you know, like they're interacting. He was Speaker 5 (16:42) obviously an AI psychosis or it was performance art. I don't know. I don't want to like, because I don't think we ever got the resolution to this. No, I don't think we ever. We kind of memory hold it, but I was thinking. back and I was like, oh, is that like the epilogue of the movie that there are people who go through like a psychosis? And just have you guys seen this? Do you remember this, Scott? Speaker 1 (17:04) Yeah. Yeah. I remember seeing it. I also remember when, you know, agents really started to take off about a year ago and you could get you could get a lot of work done in a short period of time. And then, you know, the way that we get work done. you know, traditionally is that you, you kind of have, you have like, as humans, we have days, you know, humans go out and do things. They come back, you, you have a sync that your agents can do things like way faster. And so folks who are, you know, really taken by this and sure. I, I, I was a person who was, you know, obsessed with like, how much work can I get done in a short period of time? to where like, I'm going to sleep talking to my agents, I'm waking up in the middle of the night talking to my agents, the first thing I do in the morning is I wake up and, you know, see what work they accomplished, etc. And I can see how it can go the other way. If you're seeing signs, you know, just because it's very different than normal life, you're getting this, you know, intelligent interaction that's happening, but you're getting it at all hours of the day. And in in many, in many ways, it's like, too much of a good thing can sort of on this, it'll lead down a path like this. It's Speaker 5 (18:09) so interesting, Scott, that you mentioned the cadence of this and the velocity of it. That is a way that people tend to disconnect from reality or get psychosis is they're doing too much of something. And if like speed is another one of these things that can send people on this, where they just become manic, right? The human brain is not meant to process that much information to deal with that many permutations. That's where the doom scrolling concept comes out. You can, and if you go back to science fiction, like if you look at a clockwork orange, how do they decide to torture the subject in there? They keep his eyes pried open and they make him watch television, like in an MTV stuff. Speaker 2 (18:51) Violent, Speaker 5 (18:51) troubling images. Yeah, yeah. Over and over again. Like, I think there's, the thing that concerns me is people who've been one-shotted, which I think Jeff, if it could have been performance art or. He could have been being impacted by this. I think the one-shotted phenomenon could lead to people doing dangerous things with AI, right? People who are susceptible to this kind of... phenomenon. Speaker 3 (19:14) I'm also really curious on the whole outsourcing of decision-making to AI that is happening so much right now across both consumer and businesses. I was with a friend who went on a date, and basically both he and the person that he was on the date with have exclusively used ChargingGPT to communicate with each other before the date. And when they met at the date, they realized that they absolutely hate each other. And the whole communication... predating the date was like extraordinary. So it was essentially two AI agents talking to each other and like people just outsource some of the most important decisions to AI. And what sort of consequences does that have? That would be really interesting to see. That's insane. Speaker 1 (19:54) And if you're a person who... analyzes things, you know, who builds things with software, et cetera. This is a new era, right? Because you can analyze as many things as you want. You can build as many things as you want. So you can get that dopamine hit. If that's something that gets you going, you know, then you can get as much as you want here. And if you're swiping on all the dating apps and communicating with a whole bunch of people, like you can be the slickest person, you know, to all of them, et cetera, get these dopamine hits, et cetera. I think people are just going to have to figure out how to moderate. That's it. It's just like with anything, you know, you, you're, you have to figure out your own limits and there will be some people who, you know, end up, you know, in a spot that has gone too far. Some people that are sort of laggards and adopt in adopting it. And then there'll just be like a new norm where everybody kind of figures out what the moderation is. Can Speaker 2 (20:43)