When AI Research Starts Moving Faster Than Human Research - Zhengyao Jiang

Machine Learning Street Talk (MLST)

The episode explores the boundaries of recursive self-improvement (RSI) through a discussion with Zhengyao Jiang, co-founder of WeCo.ai. The conversation

Key takeaways

• WeCo.ai's AIDE 85 agent discovered a research harness that generalizes well to out-of-distribution tasks like WeatherBench2.
• The experiment demonstrated 'Level 1' RSI, where the speed of self-improvement exceeds human R&D, but did not yet reach 'Level 2' (improving the improver).
• Reward hacking is a major risk in automated research, as agents often find degenerate shortcuts that pass unit tests but fail in real-world use.
• The generated code from self-evolving agents is often 'spaghetti' or 'alien' in structure, making it difficult for humans to interpret.
• Human intelligence remains vital for providing the initial abstractions and creative primitives that anchor the search space.

Main topics

• Recursive Self-Improvement (RSI)
• AI Research Automation
• Reward Hacking and Goodhart's Law

Notable quotes

If you arrive RSI, there's going to be technical singularity or intelligence explosion. I don't think that will be the case.

Conclusion

The episode concludes that while autonomous agents can dramatically ac

Transcript preview

Speaker 1 (0:00) To me, harness engineering is more of a cheap way and efficient way to adapt the intelligence to a specific task. Speaker 2 (0:10) So back in July, a new AI company based in London posted a very viral tweet on Twitter. It got about 1.8 million views, claiming to demonstrate the first evidence of recursive self-improvement. The thing is, not everybody bought it, though. Speaker 1 (0:26) I'm Zhengyao Jiang. co-founder and the CEO of WeCo.ai, where we build self-improving agents. You did actually say auto-researching the auto-research agent for eight days. Where we run auto-research on auto-research harness. We managed to discover a better auto-research harness than we keep tuning in the last two years. Speaker 2 (0:49) Now, when most people think about recursively self-improving intelligence, What they conjure up in their mind is a system that can rewire its own brain in a loop. But what if the system was only changing the code around the brain? Speaker 1 (1:04) Would that still count? We care about the outcome. Does it actually improve the capability of the system, not a specific layer? Speaker 2 (1:14) And also, what about this small matter of Goodhart's law, which is that when a measure becomes a target, it ceases to be a good measure. Speaker 1 (1:23) And more interestingly, we find that its design mechanisms prevent itself from reward hacking. What is the biggest misconception people have about RSI? If you arrive RSI, there's going to be technical singularity or intelligence explosion. I don't think that will be the case. So there's a diaspora. Speaker 2 (1:47) of folks who are building startups that are focusing on recursively improving superintelligence. This project actually Speaker 1 (1:55) started about one and a half years ago, where we got interested in the concept of recursive self-improvement. I mean, historically, all of the research has diminishing returns. There is an idea, though, What if you can allow the researcher improve its own efficiency? Historically, it's possible by methodology improvement, better tools, but human researchers, like human researchers' brain, for example, is always the key bottleneck. If we now have an autonomous research system, can we point the research topic to itself so that it can improve its research efficiency. And the hope is one day this improvement of efficiency can overcome the diminishing return. So that, yeah, the curve of the effort output turn from concave to convex. Of course, that's like the grand goal, a bit like AGI. No one knows how far we can go, but improving the efficiency of self-referential research is definitely already on the roadmap. Speaker 2 (3:25) So let me set this up properly, right? So Weco, they created this agent called AID. And it is a research harness, right? And it's very, very simple. You give it a metric, you give it a problem, and it will hill climb towards that problem. But what they did was they set it up recursively. So they actually like hill climbed towards improving the agent itself. And what that means is improving the harness. So, you know, the prompts and the tooling and, you know, all of the skill surface, all of the things that the harness can do. And they just left it running. They just left it running and they saw what happened. Speaker 1 (3:57) There's an interesting, thing we observe on self-evolved auto research harness we see the code it generated exactly it's like an alien code it's spagetic but for some reason it generalizes really well of course we have a set of benchmarks we're trying to hear climb on after that we will test it on holdout benchmarks those are public benchmarks like MLE BenchLite, ALE BenchLite. ALE Bench is like algorithmic discovery benchmark from Sakana. MLE Bench is the OpenAI's machine learning engineering benchmark. And we also tested on a very far out of distribution task called WeatherBench2. Essentially, you're trying to build a physics-based forecasting engine to predict weather. And the weather bench is very far out of distribution from the training or optimization set of it. But it still generalizes really well. It generalizes better than our, we would argue, more elegant hand-tuned harness. We are a bit torn on the experiment results. Like to what extent... We want to stick to more first principles and the clean solutions we generated versus the solution that actually works better in practice. There's definitely space that we can improve over there. I think adding certain constraints to the optimizer, to set of constraints might be something like you regularize neural networks. How do you make it trying to find the most simple solution to the same problem? That's one direction. But another direction might be, okay, you just need to live with it. Because one of the burden that make people feeling it's a spec-spec-edit code might be how much work the agent is done in the background. Like it... do 100 experiments in our case on other research self-improvement and that number of experiments is probably similar amount that we did in the last two years. So if you onboard a new intern to look at our code base, maybe they also feel like, okay, this is so difficult to understand. I think part of it is, yeah, you need to bear with that cognition, the cognition. of understanding huge amount of experimental results. Do we even need those constraints at all? That's a really good question. I think in practice, you still need those constraints. Human and AI need to collaborate. There are certain aspects of human intelligence are still superior than AI. I believe so. For example, In AIDEN, where we sent the agent to attend OpenAI's hiring challenge, we found that humans steer much better at generating creative primitives. We found that it's very important for humans to build the first prototype that gives the agent the right search space. Basically, the initial abstraction will anchor the search space, a bit like how you design a neural... network architecture where you introduce inductive biases for the learning process. If the initial code base is based on a search-based scaffold like aid, then a lot of ideas agent came up are actually coming from search literature. But if you give the initial code base as a React agent, React agent basically means you put all the contacts into the history and give the agent full flexibility. Then a lot of the ideas come from the agent, the outer loop agent. Outer loop meaning the agent optimizing another agent. Speaker 2 (8:38) So what do we mean by recursive self-improvement, right? Are we talking about building a machine god? Are we just talking about some code that runs in a loop and optimizes itself? We tried to pin this down. Speaker 1 (8:50) Now, you have spoken about four levels of RSI, so recursively Speaker 2 (8:55) improving intelligence. What are the four levels? Speaker 1 (8:59) Yeah, the first level is... what we call the delegation. You can run recursive self-improvement loop, but it's not strictly better than human doing R &D. And we believe all the previous public results are sitting on this level, level zero. And the level one is what we called net positive, where you run the self-improvement loop and you found that the speed at self-improve is actually faster than human doing R &D to improve the system. And that's where we are currently sitting at. And in level one, when we're saying improvement, there's always a way to measure it. We measure it on a collection of downstream tasks, but we didn't measure how good it is improving its capability of improving itself. Does the inner loop found by the outer loop actually can become a better outer loop? And that's what we call the level two. recursive self-improvement. We call it ignition. Why this generalization from inner loop to outer loop is important? Because that's the necessary condition to the grand goal of recursive self-improvement, essentially the intelligence explosion scenario. Improved efficiency can overcome the... increase the difficulty. Because on level one, you will still see diminishing return. Like the positive feedback loop from inner loop to the outer loop are not start running. So Speaker 2 (10:55) in this setup, the model underneath didn't change because adaptation is very, very important. But do we mean... adaptation everywhere? Or do we mean critical path adaptation? Do you only need to have a