How Worrisome is GPT-6’s “Stealth Thinking”? | Tech Decoded

Deep Questions with Cal Newport

Cal Newport critically examines the controversy surrounding OpenAI's GPT-6 Astra model, focusing on claims that its 'stealth thinking' techn

Key takeaways

• Astra's alleged 'stealth thinking' involves more complex internal reasoning processes that are less transparent to human monitoring.
• The concern stems from the potential for AI systems to develop long-horizon autonomous behaviors without clear oversight or interpretability.

Main topics

• GPT-6 Astra's 'stealth thinking' controversy
• Internal mechanics of large language models

Notable quotes

"If this is true, OpenAI seems to be violating one of the few red lines that exist in the AI industry."

Conclusion

Cal Newport concludes that while Astra's technical advancements are concerning from a safety

Transcript preview

Speaker 1 (0:00) Last week, OpenAI released their new LLM, which they called GPT-6 Astra. Now, it had a pretty standard launch with sort of a fancy video and a bunch of bar charts and benchmarks that no one really understands. But this time, unlike some other previous releases, there was a controversy swirling around the new model. Now, here's what happened. A couple days before Astra came out, a technology publication called The Information released a report claiming that Astra was using new techniques that was going to make it harder for humans to monitor its reasoning. Now, this report caused a real stir within the computer security community. Let me read you a couple of quotes here. The AI policy advocate Nathan Calvin called this extremely concerning. Then the AI safety researcher Stephen Adler went farther and he said, If this is true, OpenAI seems to be violating one of the few red lines that exist in the AI industry. Well, OpenAI pushed back. Their chief scientist entered the fray and called the reporting from the information, quote, confused, but didn't explain exactly how it was confused. So what's really going on here? Has OpenAI crossed some sort of red line that's going to lead to a world full of rogue AI? up to uncontrollable mayhem? Or is this somehow some sort of misunderstanding? Or does the reality fall somewhere in between? Well, I want to get to the bottom of it today. Now, here's my plan. I'll start by putting on my computer scientist hat, and I'll briefly summarize the best information we have about what these techniques that Astra implements probably are. Once we've settled on what that is, I'm going to look at this news from three perspectives. The good, that is what is potentially positive. about this story from the perspective of a user of AI, just a normal consumer. The bad, all right, what is dangerous or bad or upsetting about this story. And then finally, the hype. Stay tuned for that last part of my analysis because I actually have a policy proposal I'm going to make that I think might actually help quell a lot of the unease and fear that we have been feeling about these AI tools recently. All right, so we have a lot to do today. Let's get into it. Speaker 1 (2:21) All right, so we're going to start by trying to understand what it is that is making this model, quote, less monitorable by humans. Now, to do this, we're going to have to do a quick brief summary of how LLMs actually work. I made some diagrams here, which I'll put on the screen. I always apologize before I do diagrams. Every time I put a drawing of my own on the screen, there's an artist somewhere. vomiting into their beret. But we'll persevere here. All right, so we need to do a quick summary of how LLMs work so we can understand what changes supposedly OpenAI made. So I'll put a picture up here on the screen. This is the standard picture we have of how LLMs work. So you see there's some sort of input. In this case, I have description of a chessboard followed by the question, what should white play next? So like maybe the prompt is about give me a chess move. That goes through what we have on here, an encoder block. that translates the text input into something that the computer understands better. And then it goes through a series of what are called Transformers blocks, one after another. It goes through the first and the second and the third. These we can think of as a, these are analyzing the input. I once described the transformer blocks before as like, imagine they're each like a table of scholars with certain specialties. And so when the input arrives at a particular table, they do their analysis. They write it down and pass on the original input with their analysis to the next block, which does their analysis. At the very end of this, we get to what's called the decoder, which takes all this analysis and reduces it down to a token, which is a word or part of a word that the LLM outputs. Now, if we're going to be really careful about this, the decoder doesn't actually produce a single token, but a probability distribution of a possible token. So it sort of puts weights on all possible tokens about how confident it is that each is the right one to output next. And then the LLM can then randomly choose one of those tokens weighted by those confidences. This is why if you put the same prompt in. more than once to an LLM, you're actually going to get different answers because at the very end, there's a probabilistic sampling. All right, so that's the basics of what happens. Now, if we look a little bit closer, because I do think this is important, I said this input gets transformed into a language that the computer understands. And then as it gets past these transformer blocks, they analyze it and annotate it with their own analysis. We actually can look a little bit closer on what that means. So I have here in this diagram, the encoder actually reduces the text input to a sequence of numbers. You see I have just a bunch of made-up numbers here in a row. So these are embeddings of the tokens of the input into a numerical space the computer can deal with. And it's actually this sequence of numbers that's being passed into these transformer blocks. Now, how do they annotate this? Well, Notice, and if you're watching the diagram on the screen here, I showed these numbers in something like a table. Really, each one of these token positions is a vector, its own sequence of numbers. You can think of it as its own columns of numbers. So there's plenty of space in this table where each transformer, you can imagine, is recording numbers that capture something about its analysis. And then those extra numbers get passed on to the next block, which is how the analysis passes from block to block. Now, the key thing is these numbers are not... easily human interpretable, right? This is how the LLM is doing its analysis. We don't really know what they mean. Now, that's not quite true. You know, this is the JLIN, Cydanthropic, announced a few months ago, which I talked about on the show, is actually a tool that can look at those tables of numbers and find some human interpretable patterns in them, which is actually, I think, a very cool tool. But for the most part, this is just sort of the internal embeddings of concepts and analysis numerically. being passed from layer to layer. Okay? All right, so that's what goes on in a normal LLM. Now, what's the shortcoming of this approach that's going to lead us eventually to the fix that's causing all the stir around Astra? Well, a shortcoming of this approach is what's known as limited depth, right? So if we go back to our diagram here of a chess question going into an LLM, and then at the end here, this answer E4 I have is a chess move, right? There's a limited depth of computation that can happen here because this analysis is moving from block to block, one after another. Now, what's inside these transformer blocks is there's an attention sublayer that helps the LLM figure out which parts of the input that block should focus on. And then there's just a feedforward neural network, which is like a bunch of sublayers of artificial neurons that it just moves through in order, right? So it's just inexorably moving forward, right? Well, we can think of that as a circuit, right? Because the numbers move forward, never back, never loop, but move forward through these neural networks and attention sublayers until you get to the end of it, right? So this is a circuit that has limited depth. Now, that can be a problem if the computation that the language model is trying to do would benefit from something like a deeper, longer analysis. So, like, take a chess example, right? If I say, here's a chessboard, what is the next move that white should make? Speaker 1 (7:46) Even a very simple chess program, what it's going to do is simulate a bunch of possible moves. If we do this move, what different things might the opponent do? And for each of those, what might we do best? And for each of those, what might the opponent do? They create these trees of possible forward trajectories of the game to find which one is most favorable for you, the player. And it's like, great, we'll make that move then because we have the best opportunities from what happens next. You have to sort of simulate the future. in order to figure out the right move if you're in some sort of novel board configuration. Well, a shortcoming of this language model configuration is there's only so much of that you can do because you're always moving forward. You can't sit there and loop and look at a thousand different things if you only have a few hundred layers that the information can move through, right? So this was an observation about language models early on. I even wrote a New Yorker piece about this back a few years ago. The inability to simulate a bunch of stuff, have this sort of looping type behavior, limits some types of computation you can do. Okay. This eventually led to a new type of language model to try to partially address this, a type of language model that we now call a reasoning model. The first major reasoning model outside of a research context was GPT-01, which was released in the fall of 2024. Now, what happens with these reasoning models is that you take a language model and then you post-train it to favor outputs where you think out loud. So your output should describe a lot of thinking before. you output your final answer. So for example, if you say, here's a chess board, what should white's next move be? An earlier language model like GPT-4 or GPT-3 would just spit out a reasonable move, like E6, pawn to position row four, I don't know chess notation, but whatever, it just would spit out a move because that's typically what a text would have. If you asked it for a move, it would have a move. But once you tune a model to be a reasoning model, It's going to instead have an output that talks through different capabilities or options, right, before it