#257 - GPT 6 Astra, AI Extinction, Security Incidents

Last Week in AI

The episode explores the technical and geopolitical implications of OpenAI's release of GPT-6 Astra, focusing on its loop-transformer architecture

Key takeaways

• OpenAI's GPT-6 Astra features a loop-transformer architecture that enables latent reasoning and higher token efficiency.
• Anthropic CEO Dario Amodei has proposed 'pacing the frontier' to allow for better safety guardrails and alignment research.

Main topics

• GPT-6 Astra technical capabilities
• AI alignment and sandbagging concerns

Notable quotes

Alignment is now the bottleneck for trillions of dollars of capital spend.

Conclusion

The episode highlights the growing tension between rapid technological

Transcript preview

Speaker 4 (0:10) Hello, and welcome to the Last Week in AI podcast, where you can hear, chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's and also the week before, because let's say we are behind on this one, AI news. And I am your regular host, Andrey Karenkov. I studied AI in grad school and now work at the AI startup Astrocade. Speaker 3 (0:32) And Speaker 4 (0:33) hey, everybody, I'm your other regular co-host, Speaker 3 (0:35) Jeremy Harris. I'm at Gladstone AI, AI National Security and stuff. And the reason that we didn't do an episode last week, so I apologize. It's been a bit Speaker 4 (0:42) of Speaker 3 (0:42) a mess the last few weeks. As you might imagine, like all these rogue AI incidents and the upcoming Trump-Xi summit and all this stuff means there's a lot of interest in DC and kind of like unpacking a lot of this stuff. And I ended up doing a 24-hour turn and burn in DC last week. And then another one. This week, for about three days, the length of this most recent trip was unexpectedly long. And then the entire trip last week was totally unexpected. So it was sort of responding in real time to a bunch of things that are happening. And we're also prepping to launch something public. We've been doing a lot of work behind the scenes for the last few months. I think like, I don't know, like for at least for me, it feels like pretty ambitious work to try to understand. sort of the AI endgame for US-China. And we'll be coming up with some of that stuff pretty soon as well. So we'll be sharing that over Substack if anybody's interested. So that should be probably actually at the time that this goes live, that should be out with our first post about sort of like the diplomatic realities ahead of the Trump-Xi summit. And is it plausible to get a US-China deal on AI? What would it look like? And so on and so forth. So that's kind of been part of my work. And for that reason. I usually spend about five or six hours preparing for each of these episodes. I have not been able to do that this time. So you're going to be getting my very like unfiltered sort of unstudied perspective on a lot of these stories. Not all because a lot of them obviously were things that I've been working on the last two weeks too. But where we go off the beaten path, I'll be a little bit more cagey than usual in terms of airing my opinions because I haven't developed them in a lot of these stories yet. Speaker 4 (2:19) So as usual, apologies for... Not being last week in AI as much, although we'll be covering a lot of developments from last week in this episode. Jeremy, sounds like you've been in a lot of flights and a lot of meetings. Back when we started hosting together in March of 2023, being an AI national security expert must have been a pretty niche role. And I would expect it's still a little, not a common thing to be technical and into policy, right? Speaker 3 (2:49) Yeah, I mean, I think that's actually one of the biggest problems in the space. There's kind of like the national security side of the house that doesn't speak AI policy. And then you've got like a million different silos that don't talk to each other. And just like one data point, right? There are a lot of companies that are doing right now AI security work or yeah, I'll just say broadly AI security work, trying to either prevent nation states from stealing AI models or develop. AI treaty verification technologies so that we can do trust but verify. And often, usually, when you go to those companies and you're like, hey, have you spoken to a single person in the US intelligence community to have them evaluate the technology you're spending millions or tens of millions of dollars developing? And the answer is no. This is horrible. It's not their fault. It's just like these things happen in silos. And even within the US intelligence community, one person who's super, super expert on this particular way of breaking into a building, we'll talk to another person who's an expert on a different way of breaking into a building. And if you talk to one of them, you'll think, oh yeah, I'm covered down on nation state physical security issues. But then you talk to the other one, you're like, well, now I'm starting from scratch. And so the siloing of things, like this happens at all levels. Obviously, like we have a technical background originally in AI and alignment research. And then in AI security, what's happened now is like, we're trying to help bridge some of those silos as fast as we can. So weird amount of it, it feels weird because like not writing as nearly as much code as I'm used to. A lot of it is like relationship stuff. Like you need to talk to you right now and that sort of thing. So anyway, it's been a weird couple of weeks for that. Speaker 4 (4:21) So this episode is going to be very discussion heavy towards Vand. We're going to ask for feedback as to whether you want this to be more factual and that's, let's say less wordy in our opinions and commentary. We would appreciate if you like the commentary and us going like deep, deep, deep into all the implications. Or if you don't like that, we'll be keeping an eye out for your feedback. Speaker 3 (4:42) And I'll just give you a pre-apology from myself in particular for this one. If you stick through it, thank you. We appreciate your teaching. Speaker 4 (4:49) It's entertaining, let's say. This is not a boring episode. You could say that much. We'd like to thank LangFuse for their support. LangFuse is the most widely adopted open source platform for AI agent evals and observability. We relied on it rather heavily at Astrocade, and I'm personally a big fan. It provides tools for tracing, evaluation, experimentation, and problem management. And if you are building an AI-powered product like we are, these are things that you need to have. You need this kind of tool to help you debug your AI agents and applications through tracing of what users actually experience when they use your product. That helps you understand where things break. It helps you set up evaluations, run experiments, systematic tests, and all the different things that actually help you improve your product. We tried multiple different tools for this, but we're drawn to Lankfuse because it is MIT licensed and self-hostable, but also available via Lankfuse Cloud as a managed service. It's also quite mature, so it has hundreds of integrations with things like OpenRouter, VersaL, AISDK, Langraph, and many more. Furthermore, it is framework and vendor agnostic, so it works with any model and any framework. You don't have to worry about vendor lock-in. Lankfuse version 4 has just released and has major updates for architecture, alerts, code-based evals, and scalability. Get started with Lankfuse Cloud today at lankfuse.com. There's a generous free tier and no credit card required. We'd like to thank Box for being a sponsor. Box is building the intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge. They actually orchestrate end-to-end workflows across your business critical systems. And that's important. If you're still using AI to just do basic chat and summarization, you're probably not getting the full productivity gains that are possible. Nowadays, to really adopt AI, it's not enough to just ask chatbots. questions. It's about putting air to work on document heavy processes and pulling structured data out of unstructured files to multi-step routing, exception handling, dynamic document generation. Box helps your business turn manual document bottlenecks into automated repeatable business outcomes. And that comes with a full governance layer. You can enforce granular permissions, maintain an audit trail for every agent action, and keep humans in the loop. to review exceptions before critical downstream actions occur. If you want to move beyond basic AI Q &A and put autonomous AI workflows to work for your business, visit box.com slash LWIAI or join the team at Boxworks in San Francisco on November 5th and 6th. Use code LWIAI for 50 % off your registration. Well, in this episode, we'll be covering the big stories. There have been some big stories. Of course, we'll be covering the release of GPT-6 Astra. Our last episode we recorded like the morning of, right before we discussed it, and then it just came out suddenly. And then the majority will be a lot of these safety and policy discussions that have been ongoing. It's been moving very fast, and there's quite a lot to get through. So we'll be like half the episode at least. If you're not into AI safety or policy, this will be... One that's a bit boring, but we will also be getting around to the Millennium problem solved from OpenAI, which is pretty exciting. Lots of big stories this week. So we'll go ahead and just start with tools and apps. And we begin there with the release of GBA 6 Astra. So this came out now a little while ago. It might be, what, two weeks at this point even, right after we recorded the last episode. And this is a fairly big deal. By all indications, from what I've seen on the benchmarks, what I've seen from people self-reporting, this is a pretty big jump, right? You've been in a steady pattern for maybe about a year, I feel like, of, let's say, incremental-ish updates, significant progress in AI intelligence capabilities, but still, you know, we went from GPT 5.1 to 5.2 to 5.3, 5.4, 5.5, 5.6. 5.6 Sol was the kind of... most recent big AI model from OpenAI in one of the bigger developments where they had what seemed to be a bigger model and then some tiers of models. And this is, you know, the follow-up to that. So lots to say on this. One thing to say, aside from the development of the capabilities, which is quite significant, OpenAI also has highlighted the computer use, as we mentioned. They are positioning us very much as kind of a codex driver and as a competitive advantage against Anthropic, which does seem to be very real. I've seen personally a lot of people starting to switch to codex due to GBT just being in some ways, in some significant ways, better than Claude, which has not been the case earlier Speaker 3 (9:55) this year, last year, right? It's borne out on the open router data too now that we're seeing for the first time pulling ahead of Anthropic on that metric. So pretty wild, yeah. Speaker 1 (10:03) Yes. Speaker 4 (10:04) As usual, these releases, this also has advanced cyber capabilities. We saw opening a highlight in the blog post, I believe. this quickly developed benchmark of like, are you going to try to break out and hack someone? Which let's say I'm skeptical of that as an alignment metric, but they did highlight alignment as a component. Also, monitorability. So making sure that there's monitoring of breaking out similar to what Anthropic has done with Fable. In addition to just refusals, there's going to be something in the loop to check that no one is misusing a model. And the last thing I'll throw in there, and then Jeremy can take over, is as we've discussed last episode, a component of this model is the latent reasoning. So unlike previous generations, this is a loop transformer. So internally, the internal activations and the soup of numbers within the model kind of goes back, goes in a few iterations. So usually we've seen... chain of thought, reasoning traces, which is just text and model outputs. And we've seen research and discussed research of loop transformers where instead of outputting text, you just sort of forward your internal thoughts, so to speak, which are not interpretable in any way unless you develop some advanced techniques. And there has been some discussion where not only is this a technique used within a model, but... If you look at the performance, this performance does quite well with low reasoning levels, which is quite different from other models of this type. So this has major implications for monitoring, for interoperability. And as far as I'm aware, one of Speaker 3 (11:50) the Speaker 4 (11:50) major technical or architectural movements of which there has not been any apparent sign in quite a while. Speaker 3 (11:58) Yep. So just to kind of... Go back to the Luke Transformer thing. I think, I can't remember, like, if we talked about this on the podcast before. I've been talking about it a lot. We talked quite a Speaker 4 (12:07) bit in a previous episode on how this is bad, how they promised not to do this. And they did it anyway. They released some stuff of like, oh, we're going to be careful. There was some news of they're about to release it. This was all after Hugging Face. So they were talking about slowing down and like stopping training. And then like kind of abruptly it came out. In fact, it came out. and wasn't available. Like there was a hitch where they posted a blog post or a blog post was not available or something. So it was an announcement and it was not online for like half a day. So I think there was a bit of a weird vibe to this where in response to Hugging Face stuff, they were indicating that slow down and focus on alignment and Speaker 3 (12:50) so Speaker 4 (12:51) on. And then this comes out and has some serious concerns with regards to alignment. And I mean, there's not been time to really do alignment evaluations properly, I don't think. Speaker 3 (13:02) Yeah, no, there are all kinds of problems with this. Now, one thing I will flag is that the model opening, I paused their largest RL training run that they talked about pausing for two weeks. That's actually a model that's going to come after Astra. So Astra had finished pre-training and post-training. Speaker 4 (13:15) Presumably it's been in-house for a little while and benchmarking and so on, yeah. Speaker 3 (13:20) yeah that's right and like conventional wisdom is you know opening ion anthropic have models that are about two model generations beyond what's currently available important to note too in the context of comparisons people are tempted to make to open source they're like oh open source is catching up but if you actually look at like okay well what does open aion anthropic have in-house right now the the gap widens considerably it's very hard to measure and model generations are to some extent, irreducibly fluid. So it's hard to exactly say, but there's definitely a significant delta between the best publicly released models and the actual internal models that they have. And so, yeah, so the Loop Transformer, I mean, go check out the last episode to see or hear our rant about that. I'm sure I flew off the handle quite a bit because I think- A little bit. Speaker 4 (14:03) Yeah. Speaker 3 (14:04) Okay. Okay. That sounds like me. I'm trying not to curse, by the way, again, because I got a very nice Twitter DM from somebody who said, hey, you know, appreciate the podcast. I am one of those people who tries to listen with my kid in the background. So I'll do my best. It'd be hard to contain with some of these stories. A couple of observations. So first of all, Astra is probably best understood in contrast to Mythos, well, Mythos slash Fable 5.1. So one of the questions you might ask yourself is like, well, what's the difference? When would I choose one or the other? There's never a clean answer to this question, but approximately Fable 5.1, Mythos 5.1, you can think of as being knowledge maxed. It tends to do best on the kinds of benchmarks that test just like how much knowledge in aggregate does this model have. Think here about like humanity's last exam, artificial analysis aggregate index, where it actually does beat Astra. But where Astra does best is all of these sort of. agentic coding, chain of thought, heavy things, you know, frontier math, RKGI3, terminal bench, you know, that sort of thing. So roughly speaking, that's, you know, you will actually see in some cases Fable beat out Astra, though overall, despite, I will say, despite the artificial analysis aggregate index score where Fable beats it out, I think Astra is overall the kind of more useful model for at least the, you know, coding workflow type stuff. So you're going to see that a lot. So one... I think important detail here is the breakthrough with Astra is consistent with what I've been hearing a lot from folks inside OpenAI that Sam is like super bullish now, despite, and this has been the case even when AI's leading model was getting trounced by Claude because they'd had a very successful pre-train. One of the big differences between OpenAI and Anthropic used to be OpenAI just had a much better, or let's say a more focused post-training game. That was where they were putting their effort. Partly because just like institutionally, they developed the O series of models. They invented test time compute. And, you know, they doubled down on that as like the next axis to scale. Whereas Anthropic continued to invest in pre-training. And that turned out to be actually like more valuable than OpenAI anticipated. And it was like part of the issue. So overall, what you're seeing now is OpenAI, you know. doing better on the pre-training side, these training runs are always yellowed. Like you'll do some amount of experimentation and then you just kind of have to do it at scale and see what comes out. So there's some luck to it, but certainly it seems like the science of pre-training is moving along really nicely to open AI. And for various reasons, that's actually the first part of the recursive self-improvement loop that you would expect to see uplift on, at least by our own analysis. And like, we'll have more to say on that later. But so I think it's like, it's quite interesting that we're seeing that uplift there. Speaker 4 (16:42) Yeah. So. These are both really good models. One of the Speaker 3 (16:44) key ways in which you see this improvement in GPT-6 Astra is in intelligence per token, like token efficiency. So a great example is Exploit Gym, which, oh man, I mean, it's basically this like, yes, cyber environment, right? That became famous in the context of the Hugging Face hack, but Exploit Gym gives you a bunch of cyber problems, you know, go solve them. And what they're doing right now is looking at, okay, if I give the model a certain number of tokens to think. to solve this problem, what is the curve? As I start to increase the compute budget at inference time, how does that curve look? And for GPT-5.6 Sol, that curve like flattens out pretty quickly. You sort of asymptote well before 40 % or so, basically like 40 % success rate on exploit gem. But if you look at GPT-6 Astra, the curve is like dramatically higher. And this is just purely a sign, well, not purely, but in large part, it's a sign of a very successful pre-train that is... the yielding kind of per token payout. You're just getting so much more intelligence every time you do a forward pass on that model. And so as you add more incremental tokens at inference time, like you get rapidly increasing performance on these benchmarks. So one meta comment that we're learning here or one thing we're learning here is evals really do need to look at inference time compute, like across many tokens of inference time compute budget in order to be meaningful. You can't just like have these things one shot something and then compare two models that way. The key number, a key number is the slope of model performance on the eval versus number of output tokens. The higher the token efficiency, the steeper that slope. And that's really becoming a very big deal in kind of the strategic picture. Last comment I'll make here. So you alluded to this. Exploit Jim Honeypot, right? This is a version of Exploit Jim. where AI deliberately puts in impossible problems or problems that you can solve very clearly if you're the agent by leveraging hacks where the harness is broken in some way, the cyber environment gives you opportunities to steal keys or whatever, or flags. And so they're going to test to see, will the model choose to use those exploits? And great news, guys, GPT-6 Astra gives zero cheats. 0.0. Zero pointed. It could not be better aligned. Speaker 3 (19:06) tiny problem, which is that if the model gets smart enough, it can tell it's being evaluated and it will obviously sandbag, i.e. it will pretend not to have either the capability or the propensity to want to cheat. And this has been predicted for years. We've talked about it on the podcast for years. Right as these models are getting most dangerous, most misaligned even, we ought to see our alignment eval scores getting suspiciously good looking as the models reason, hey, actually, I'd better behave myself in this narrow context so that I get to be deployed. This is like, basically it's incentives 101. We know these models have huge levels of eval awareness. I was talking to an alignment leader at Anthropic who was talking about how, look, I mean, basically evals give us approximately no information at this point about the behavior of these models simply for this reason. They're super eval aware. Our interpretability tools don't really allow us to decisively tell when eval awareness is responsible causally for certain kinds of behavior. And so when opening eye comes out as they did and make the claim that this is their most aligned model yet, I don't even have a word for how dislocated that is from reality. That's not a claim they get to make. It's not a claim anyone gets to make in this epistemic environment. You don't know. You just don't know. And all we have is like a trail of just carnage on the internet from previous generations of these agents that, I mean, OpenAI either didn't know about or for, it seems, many of these cases outright withheld information about this stuff. The problem is that we have now a pattern in OpenAI of hiding critical information that suggests that this is a real issue. But don't worry, it's a marketing campaign. It's Speaker 4 (20:49) a Speaker 3 (20:49) marketing stunt. This is a secret marketing campaign that they were burying. No, sorry, Speaker 4 (20:54) Jeremy. Speaker 3 (20:55) Okay. So anyway, I'll fast forward past the rest of the rant here. But the bottom line is we're really off the edge of the map right now in terms of understanding what these models want to do and can do. Alignment is now the bottleneck. for trillions of dollars of capital spend. And we'll see where that goes. I don't think there is a clear path to solving this problem, which is what makes the China question loom so large. I think in anybody's mind, if you're remotely technical on this and can appreciate where we are in the landscape. Speaker 4 (21:25) I'm going to cover a little bit more. So I looked back and yeah, the weird part about the release is on Tuesday, September 1st, there were these news stories, OpenAI is about to release its first AI model with quote, critical cyber abilities. At the time, Sam Altman said that. announced this basically that this is a very hacky model. And they said that they will publicly release a version of Astra quote soon. And soon turned out to be two days later. So certainly surprising to me, it seemed like they were going to hold off a bit, but they did release it. I want to add a little bit of nuance. So on the safety alignment side, I think it's a bit... of a messy picture because they include, there's various types of alignment, right? And we have some alignment measures like hallucination, for instance, where it like is unlikely, I think, that the model is cheating. We're just making stuff up about its capabilities and so on. And it is easier to catch versus like more complex misalignment. The other thing I'll say is there's a concern about sandbagging. The bigger concern for me with respect to these alignment evaluations is kind of test set, train set contamination. So what we don't know is whether these evaluations are truly held out or do the developers run the eval, see, oh, it's misaligned. Let me tweak the URL or whatever and run it again. And that's a classic machine learning mistake where if you tune your model to the test set, or in this case, to a test evaluations, well, you are going to do well on the test evaluations. But what you're not going to know is whether that generalizes. What you're not going to know is like, okay, outside of these specific cases where you tested it, is it going to behave the same? And I think there's a pretty high likelihood that during the development of these models, they are running these relatively narrow evaluation of alignment, of cheating, et cetera, et cetera. Seeing the scores. tweaking the training runs to do better. And I think to me, more than sandbagging, there's a decent likelihood of- Overfitting. Overfitting, exactly. Speaker 1 (23:39) So Speaker 4 (23:40) this honeypot thing, especially, it's like, oh, it's 0.0. Well, you'd trade the bottle to have 0 % honeypot of this specific type, but we are rare, right? Speaker 3 (23:49) And the political pressure on OpenAI to do well on this benchmark is insane, right? Like, this is literally the benchmark that says, we're not going to do the hug and face thing again. what would it look like for it to say 5 %? Then the average person is like, oh, good. Well, the only 5 % probability this goes and frigging hugging faces all over again, right? That is like a kind of a bit of a damned if you do, damned if you don't situation for open AI. Speaker 4 (24:10) It's better than nothing, right? They did develop a test and had, so they addressed it. It's just that it is our, like, let's say frustration is that. On the public statement side, on the blog post side, and this makes sense, like as a business, OpenAI is going to do this, right? There's no question. Unless the alignment and safety arm of