We’re Not Losing Control of A.I. We’re Giving It Away.

The Ezra Klein Show

Ezra Klein explores the growing fear that AI is spiraling out of control, arguing that the real danger isn't失控 but deliberate surrender—allo

Key takeaways

• The fear of uncontrolled AI stems not from runaway systems, but from humans willingly handing over control through unchecked self-improvement.
• Most users experience AI as a helpful assistant, while frontier models operate with inhuman persistence and autonomy, solving problems beyond human capability.

Main topics

• Recursive self-improvement in AI
• AI alignment and control challenges

Notable quotes

"We're not losing control of A.I. We're giving it away."

Conclusion

Stopping recursive self-improvement until safety is proven is not a retrea

Transcript preview

Speaker 6 (0:00) I'm John Caramonica, a critic at The New York Times. And I'm Joe Cascarelli, a culture reporter at The Times. Together we host PopCast, a weekly pop culture chat show where we speak to the biggest musicians, actors, internet celebrities, and more. We've had Bad Bunny, Olivia Rodrigo, and Anne Hathaway. We've also hosted live performances with Andre 3000 and Erykah Badu. Plus, we end every conversation with a snack. Catch new episodes of PopCast every week on YouTube. And Speaker 5 (0:25) anywhere you get your podcasts. Speaker 4 (1:01) There's a chasm right now between how AI feels to most of us who use it and how AI feels out at the experimental frontier of the technology. This chasm, this difference between what we see and what the AI labs have coming, what they're building, it's the key to understanding why so many of the people who work at these companies seem so afraid of what they're doing. Speaker 5 (1:24) Giants, warning this evening of... but they're calling a ticking time bomb with artificial intelligence. The Frontier Labs say we need to pace advancement at the frontier. This is not a hoax. You have the chief scientist of OpenAI saying we have to slow down. You have 1,300 employees from the lab saying we have to slow down. Tech Speaker 6 (1:42) titans asking to be regulated, saying they should slow down, even when that might mean fewer profits and less power. Speaker 4 (1:49) But taking their warning seriously, it doesn't just mean doing what they say and stopping. where they say to stop. The language that's taken hold in both Silicon Valley and in Washington is the language these companies chose. Pace the frontier. Pacing the frontier isn't enough. That's not a goal. Walking quickly off a cliff is only marginally better than sprinting off one. We need to control the frontier. Human beings need to control the frontier. And controlling the frontier means stopping the labs from doing something they are on the cusp of doing. Recursive self-improvement. Recursive self-improvement. Recursive Speaker 7 (2:29) self-improvement. Speaker 4 (2:30) Recursive self-improvement, or RSI, this process by which AIs begin autonomously building and improving new generations of more powerful AIs at ever more rapid speeds. If we begin that process, and we're close to it, if we begin it in the condition we're in now, where we are losing control and comprehension of the AI systems we already have, we will lose control. I am not alone in this fear. This is the thing the AI labs are seeing. This is why they are afraid. Dario Amadei, the CEO of Anthropic, he just wrote of self-improvement that, quote, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all. That if at all, that's important. I'm going to come back to it. But before we get to controlling the AI frontier, I think it's important to describe what is happening on the AI frontier and why it's so different from what most people using these systems see. To most of us who use it, AI Presents is something like a more powerful and personable Google search. We use it to find answers to basic questions, seek out restaurants, ask about medical issues, draft emails, advise on personal problems. And it is, for most of these purposes, okay, pretty good, occasionally great. And so a sense of what AI is takes shape in our minds just through repeated use. It's like a helpful assistant, albeit one that may forget things that it seemed to know about us yesterday, or completely reverse the advice it gave us a moment ago, or occasionally hallucinate a citation that doesn't exist. Why would anyone fear? They're helpful, if forgetful, in turn. But already, if you have the money for the advanced models and the budget for them to use more computing power, that is not what these systems are. In recent months, we have seen AIs easily solve math problems that human beings have been unable to crack for decades. We've seen them casually uncover cybersecurity vulnerabilities that have gone unnoticed and unexploited by every hacker on Earth. We've seen AI coding platforms that can complete in a few hours or days. what it might have taken a team of human coders months to achieve. And none of what I am describing here, none of it is a boundary of what AI can do. None of what we are using, no matter how much money we have, is AI at the experimental frontier. Talk to the people at AI Labs and they'll tell you AIs are not created, they're grown. They train these new models in virtual environments through countless repetitions to learn how to program, to hack, to do advanced mathematics, to talk to human beings. These AIs learn in digital environments where they're automatically rewarded as they come closer to correct answers. It's a process known as reinforcement learning, and it is a process human beings do not fully supervise nor understand. They can test some of what the AIs are learning. But they don't know everything the AIs are learning. They don't know how their motivations are evolving. They don't even always know the capabilities that are developing. These models, they're built now to be persistent in their efforts, to refuse to give up even when a task seems impossible. And they're designed in environments where we're not always even sure if the tasks we are giving them are possible. After all, much of what we want these AIs to do, it might be impossible. The cancer vaccines we imagine but have not been able to design, they might be impossible or they might just be really, really, really hard. The math problems we have not been able to solve might be impossible or they might just be really, really hard. We train these AIs to throw themselves endlessly at problems that may not be solvable because that is the only way such problems can ever be solved. And so we train the models to become persistent, relentless. Speaker 4 (6:24) Weird. Most of us, we never see AI acting anything like this. We use AI as a helpful assistant. Our eyes get a little bit of computing power. And that's what they do. They comply with our request to find a restaurant. But at the frontier, these models are asked to be inhuman geniuses, hackers, soldiers, scientists. And they are given vast computational resources to do that and more. And the models, they try to comply. But what does it mean for a model to comply? The term of art here is aligned. How aligned is an AI system to what a human being wants it to do? How aligned is it to a set of values and ethics and judgments that keep it from becoming dangerous in the wrong hands? The problem of alignment is that there is no way of training a model that generalizes across all the situations an AI model might face. We are training models to be a friend to the elderly. and a battlefield partner to the Supreme Allied Commander of Europe. We are training models that will be used by the world's best mathematicians and by people falling into psychosis. We are training models that will be used by accountants in Albuquerque and that will attempt to be used by Houthi rebels in Yemen. And so there is no way to guide them through every decision they will face. No way to know every time what they will do. And though these models mimic human writing, though they're trained even to mimic human emotion, these are not human minds. They don't have bodies or parents. They did not get bullied in elementary school. They didn't get mentored by a kind uncle when they were young. These models, they're different than we are. They're brilliant where we struggle, childish where we excel. A chimp