Noam Brown – Agent swarms, alignment, & recursive self-improvement

Dwarkesh Podcast

Noam Brown discusses OpenAI's breakthrough in solving a Millennium Prize problem using a multi-agent system of 10,000 AI agents, exploring t

Key takeaways

• Multi-agent systems enable scalable test-time compute via parallelization, though with sublinear speedup as agent count increases.
• The solution to the Millennium Prize problem was driven primarily by model strength, not multi-agency—though coordination played a role.

Main topics

• Multi-agent systems
• Scaling test-time compute

Notable quotes

"If you have four agents working on the problem, it is done twice as fast. You're basically paying a 2x more to get an answer twice as quickly."

"The effort to get a Millennium Prize problem was not due to multi-agent. I wouldn't even attribute 10% of the credit to multi-agent."

Conclusion

While multi-agent systems unlock unprecedented cognitive scale and parallel reasoning,

Transcript preview

Speaker 2 (0:00) Today, I'm chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became O1 and the reasoning models, and now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the millennium price problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours. One of the reasons I was interested in talking to you is I think you were one of the first people, maybe two or three years ago, who was... Thinking about how the reasoning models would allow us to see into the future, because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future. And I feel like you're in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now. Speaker 1 (0:47) So the way I think about it, when you plot the performance of these reasoning models with test time compute on the x-axis and... performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's the same thing with people. If you're taking the SATs, you have five minutes to go through the entire exam. You're not going to do very well. If you have five hours, you're probably going to do a lot better. The AI models are pretty similar, and they'll spend that time doing this monologue to themselves, figuring out, going through different cases, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don't want to sit around for three years waiting for a response. And so what you can do is what a lot of people do is they paralyze. They just get a team of people. If you're going to found a company, you want to get a group of people together so you can go faster. It's the same thing with these AI models that it helps to just have multiple agents working on something because they can just go faster. And so multi-agent is a way of scaling test time compute in parallel instead of purely serial. And it is like less efficient because it doesn't have, it's not like a single agent has all the context to itself, but it is like a very effective way of scaling test time compute if it's done well. Speaker 2 (2:08) Okay. I'm Speaker 1 (2:09) going Speaker 2 (2:09) to ask a bunch of naive questions because these systems, so this is an unreleased model, so we haven't publicly seen how these systems work. And so I just have a bunch of ways in which I'm confused about what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as a full-time job, stretched back to back, 130 billion tokens would be a human thinking for like 4,000 years, you know, eight hours a day or something, working a normal work week. So starting from like ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours. I feel like qualitatively, that is a super important consideration. And I'm surprised that there isn't a bigger parallelization penalty. that you can just have 10,000 agents collaborate and because maybe the agents are better at collaborating than humans might be, they're going much faster, that they can actually productively collaborate at such a big scale. Or maybe they, I don't know, maybe there is a big parallelization penalty. Speaker 1 (3:13) Yeah, let's talk about the parallelization penalty and then we can talk about the qualitative stuff because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale. Speaker 2 (3:20) So Speaker 1 (3:21) when we released 5.6, I think that was the first time that we had... a proper multi-agent system in our models. And we actually did in the blog post show some plots of the scaling performance of multi-agent systems because we have it as an option. It's ultra mode. And the default is four agents, but you can set that to higher. And in the plot, we show, okay, here's what the performance looks like on some benchmarks for one agent, for four agents working together, for 16 agents working together. And what you see, and it depends on the benchmark, but for some of the benchmarks, basically, if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern. It's like a little less efficient, but you continue to see that performance. Speaker 2 (4:11) Is it a linear serial time speedup or a sublinear speedup as you increase the number of parallel agents? Speaker 1 (4:18) I would say it's slightly sublinear, though it does depend a lot on... on the problem. So math, for example, is quite paralyzable. It's Speaker 2 (4:28) not Speaker 1 (4:28) the most paralyzable thing, but it is very paralyzable. I think web search, things like doing a deep research report, where you have to look through a bunch of sources, that's extremely paralyzable. I suspect that something like writing a novel would be very unparalyzable. So you would probably not see a big benefit from having 10,000 agents working on a novel together. In the same way that you'd probably not have a big benefit from having 10,000 people work on a novel together. So The performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is it's very hard to push that science to like 10,000 agents because it's just so expensive. Speaker 2 (5:05) You guys just did it over like a weekend. But that's one Speaker 1 (5:09) data point. We don't know how long it would take a single agent. to solve mafia or stokes. Because we haven't done that experiment yet. And maybe we will, but I mean, it's also only one data point, right? And we want to do a thorough ablation. It's Speaker 2 (5:22) actually, Speaker 1 (5:23) the experiments are just too expensive to go to that scale. So we have to do some kind of methodical science about what happens when you go to 64, 128, 256 or something and get a sense of the behavior. But it's going to be very hard to push that all the way to 10,000 and know for sure what was the... the benefit that we actually got for using 10,000 agents versus 1,000. Speaker 2 (5:44) Yeah. One Speaker 1 (5:45) thing I want to make clear is that the effort to get a Millennium Prize problem, this was not due to multi-agent. I wouldn't even attribute 10 % of the credit to multi-agent. The reality is OpenAI has trained a very powerful model. And we can get that model to operate. over very long horizons. We can get it to think in parallel. But at its core, the reason why we're able to do this is because we just have a general purpose, very strong model. And I think things like multi-agents are flashy and are new, and that probably gets disproportionate credit for that reason. But the core reason is, like, this is just a very powerful model. Speaker 2 (6:34) So the generalization... is actually quite shocking to me. The US systems, I mean, I don't know how these systems were trained, but presumably they were trained how RL training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process, I'm guessing, was the model solving anything as ambitious as a Millennium Prize problem. But the generalization was strong enough that you could have this much easier verifiable problems generalized to this much parallel effort on such a hard problem. Speaker 1 (7:04) I think that is true. First of all, we do train the model on very hard problems. So there is definitely a gap. We see if we train on some kinds of tasks, it's able to do tasks that are more ambitious than that. There is an interesting challenge that as the models become smarter and smarter, the kinds of questions we can ask them, a lot of them are too easy. Speaker 2 (7:26) And Speaker 1 (7:27) it's hard to challenge the model. And I do think that that's going to be an interesting... If I had to make... an argument for why you might not see AIs like LLMs go the same path as AlphaGo and AlphaZero and all these kinds of like game playing AIs. It might be that this kind of problem that in things like AlphaZero where you have self-play, you have an infinite curriculum. You're always playing against an AI that's like equally strong. Whereas for things like training in LLM with reinforcement learning, at least the ways that are out there right now, You give the model a problem and you ask it to solve it. And if the problem is so easy that I can just solve it in a second, it's not really learning anything. So if we run out of problems to ask it that challenge it, then that is a plausible scenario where actually like, okay, it becomes much harder to make progress. Speaker 2 (8:21) Now, Speaker 1 (8:21) I do think there are ways around that. And so we haven't really hit that as a wall yet. And I think that... If it ever became a serious problem, there would be ways around it. But it is like a plausible scenario. Speaker 2 (8:34) Yeah. And just for the audience, when you're referring to like AlphaGo or AlphaZero, you're saying like getting superhuman relatively fast after achieving human-level performance. Speaker 1 (8:42) Yeah, I mean, if you look at the trajectory of gameplay as like Go, Speaker 2 (8:45) they, Speaker 1 (8:47) within a span of like a year, went from beating a European chess champion, being like, I don't know, like number 50 in the world, to beating the world champion, to being unimaginably... orders of magnitude stronger than any human alive. And it's possible that domains like math we see a similar trajectory, but I think there is a very plausible scenario where actually that doesn't happen. Speaker 2 (9:08) Yeah, yeah, yeah. So I want to understand if in six months people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system? Speaker 1 (9:20) Yeah, I should start by talking about how these multi-agent systems actually work, which is I think a very... different way than a lot of multi-agent systems and other AIs. So... A lot of people that have approached multi-agents for things like LLMs tend to take this very scaffolded approach where, for example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task and the children work on it and then return their answer. And this seems like a very sensible setup, very sensible scaffold. It definitely helps. But there are a bunch of limitations with these kinds of setups. So, for example, if in this setup you have... a coordinator that's sending tasks to children, the children work on it and then return their answers. Well, what happens if two children are given similar tasks? Can they talk to each other? And usually the answer is no. And that's very inefficient, right? If you're given a task and it's actually really helpful to talk to somebody that might know an answer to a question that you're working on or a part of something that you're working on, it'd be really helpful if we just be able to ping them and say, hey, can you help me out with this thing? But a lot of systems don't have that set up. and adding it just increases the complexity significantly to the scaffold that you have. Another thing is like what if the child doesn't really understand or has a clarification question? So it has to choose then between, okay, do I just return and ask the question instead of solving the problem? Or do I solve the problem, just like make an assumption about what the parent wanted me to do and just solve it that way. And so in any scaffold, that people come up with, there's always limitations involved. And the approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use and figure out for themselves how to use it effectively. So we give the agents the ability to message another agent. And when it messages another agent, it is inserted into the context. Speaker 1 (11:21) Then it can do a few other similar things, but that's basically the core of it, that it can just send a message whenever it wants, just a tool call, and it can send that to other agents. And they figure out for themselves the best way to coordinate around that. And it turns out that if this is done well, you get very sophisticated behavior. And to me, it looks a lot like how human collaborators work over something like Slack, for example. When we were working on this project, it was really exciting. When we finally got it working to see these agents working on problems together I remember one one example so we give the agents a problem and then one agent says I think I've got the answer and then another agent says actually I got a different answer and then they have this whole discussion about well How did you arrive at that answer? Can you explain it to me and going back and forth and trying to clarify like what? could have been wrong in each other's reasoning. And then they finally converge and like, oh yeah, okay, that seems right. And then it just broadcasts to the other agents like, actually, I've changed my answer. I think he's right. Speaker 2 (12:21) And Speaker 1 (12:22) it just felt like a very natural conversation. It's kind of felt like when you see chain of thought for the first time that's trained through reinforcement learning and you're like, oh, this is just kind of like what a person would think if they were writing down their thoughts as they're thinking them. It kind of felt like that. So it is really cool to see this kind of behavior. And so I think collaborating with these things, honestly, it feels a lot like collaborating with a person. It's just a very natural flow. Speaker 2 (12:50) Except one qualitative difference that might become salient in the future is that... These systems will be thinking, I don't know, more than 10x as fast, right? If you just look at how many tokens per second they output versus how fast a human talks. And they're working all the time. They're not sleeping. And they're collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans. So I'm trying to think of what to qualitatively expect in a year. And is it like a sort of shadow organization that is moving 100x faster in my company than the human level is? The iteration cycle is much faster. What would take a human organization a year to do is happening within a week within this shadow organization. Speaker 1 (13:36) So will it feel foreign? I don't know. I've actually found that it's pretty surprisingly natural to work with these things right now. I think that could change. So for example, we have these ultra-fast modes that enable sampling to be 10 or 15x faster or whatever. And then, OK, it's going to be pretty hard to keep up with these things. I think the idea is these agents, when they're communicating with each other, yeah, they can go super fast, but then also they understand when they're talking to an agent versus when they're talking to a person. And their behavior will be different in those situations. Speaker 2 (14:05) So the main example that we have publicly of sophisticated multi-agent systems is unfortunately the hugging face one. And the thing I found interesting there, I mean, a lot of things I found concerning, obviously, but the thing I found interesting is just the spontaneous emergence of hierarchy. of like middle management. And it sounds like you're saying like this level of organization sort of emerges spontaneously from training. I Speaker 1 (14:30) think the details are spontaneous, but I mean, just because we're giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way, it doesn't mean like, we are still giving them a starting point. We're giving them a prior about, oh, this is what reasonable communication might look like. They also, I mean, they're trained on. A lot of human techs, they have an understanding of how humans organize and coordinate. And so that's all kind of baked in. I think it is surprising the way they're able to polish this. If you look at what it starts out at, it's not very sophisticated behavior. In fact, it's actually very difficult to get these agents to coordinate in a productive way because it's just very tempting for them to just collapse to, oh, we're all just going to solve the problem independently. Speaker 2 (15:17) Yeah, yeah, yeah. Speaker 1 (15:19) that is a local minimum that you can get stuck in. But yeah, if it's done well, they can end up coordinating it very effectively in these kinds of like very structured ways. Speaker 2 (15:28) I wrote this essay a couple of years ago called Something Something AI, What Automated Firms Will Look Like. And I was thinking about, well, if you had fully automated firms of let's say human level intelligences, what is different about the nature of AI minds that would make the organization's AI's firm different? There are a couple of very important differences. For example, that AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. And also you can spin up. or spin down an arbitrary number of instances which have the right knowledge. So if you want to hire more people, it's not like just all the schlep of finding the right talent or whatever. It's like your best talent, you can just make an infinite copy of them. Or if you don't need them for the task anymore, you like spin them down and you can just replicate the most effective parts of your organization or replicate whole organizations together, which are effective. I don't know. Where do you see these? multi-agent systems going a year from now or two years from now. Speaker 1 (16:28) I think it's a great question of like how do these things actually differ from working with a human co-worker and I think you highlighted some. Like one really interesting thing is that I mean if somebody is if you have a person and you want just like two copies of them you can't just like clone the person. Yeah. But with AIs it's actually really easy to say like okay well just fork yourself and then have those copies work on this thing and then like merge back together. I mean we already have this I think in multi-agents for Astra and and 5.6 all that when they spin up subagents, the context is forked. So it has all the context that's relevant. Speaker 1 (17:00) There are other interesting ways where the agents will differ from people. Like, what are some reasons why startups disrupt incumbents? Like, there's a few factors. One is that they're willing to take more risks. But another major factor is, like, as companies grow in size, as organizations grow in size, you see increasing misalignment. between the individuals in the organization. Speaker 2 (17:26) If Speaker 1 (17:27) you have a startup with five people and each person has 20 % share in the company, they're all highly aligned to the company succeeding. If you have a massive company with 10,000 people, you see a lot more instances where people are territorial or just care about getting a lot of headcount for their project or their team or building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted. And this is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents. Speaker 1 (17:58) It's interesting that, I mean, it's true that AI does help startups in a way. It's much easier than ever before for one person to step in and be like, I'm going to make a multi-million dollar company. It's just the AI is amplifying an individual so much. But there's also an argument that they could benefit incumbents because if the alignment problem is solved, then you don't have the issue of misalignment between individuals and the company. At least that's mitigated. The AIs, if they're aligned well, they can just be aligned to the interests of the company. And you can have 10,000 of them, and they're all going to be working as hard as if they were a 20 % share co-founder. Yeah. Speaker 2 (18:36) And it's not only that, but it's also that they are much better able to manage shared memory and context than different humans can. If tomorrow you hire 10,000 mathematicians and you're like, solve Navier-Stokes, they're not going to be able to cooperate effectively. Speaker 2 (18:55) at least not off the bat, but you can have apparently 10,000 AIs. Well, Speaker 1 (19:00) again, I want to be conservative here because we haven't measured how effective the 10,000 agents are at coordinating. We think it helped. We don't actually have good measurements of saying like, oh yeah, this 10,000 agents led to like a 2x speed up over 2,000 agents or something like that. And it is actually, I would argue likely, I don't know about likely, but I think it is very possible that, 10,000 humans are better at coordinating than 10,000 agents right now. I think that is entirely possible. I think also one trend we've been seeing is like, look, we've been working on multi-agent for a while and the early versions of this, it's very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other. And it's because like, look, Speaker 1 (19:48) When we first developed reasoning models, they weren't talking to other agents. And if now you put a bunch of agents together and say, oh, solve this problem together, they're in this local minimum