Max Nadeau on why ambitious people should start AI safety nonprofits

80,000 Hours Podcast

Max Nadeau from Coefficient Giving discusses Project Tailwind, a new initiative designed to fund the launch of technical AI safety nonprofits.

Key takeaways

• Coefficient Giving offers pre-seed grants between $200,000 and $2 million to help founders launch AI safety startups.
• Seed grants can range from $2 million to $20 million for teams that demonstrate preliminary technical results.

Main topics

• Project Tailwind funding tiers
• Talent bottlenecks in AI safety

Notable quotes

I think the most important bottleneck, maybe the only bottleneck, is talent.

Conclusion

The episode serves as a high-stakes call to action for talented resear

Transcript preview

Speaker 1 (0:00) I think the most important bottleneck, maybe the only bottleneck, is talent. In absolute terms, you can get a lot of money for ambitious non-profits. I mean, $200 million is not like... an upper limit on the largest size of a grant coefficient giving will ever give to an organization in the technical AI safety space. So I think that there's a lot of opportunity for people who are willing to like, you know, get down in the mud of these like speculative questions and are willing to entertain these sorts of out there hypotheticals. I'm hoping we're going to put out this call and then a lot of really, really talented people are going to raise their hands and say, yeah, that's, I want to work on that. Speaker 2 (0:38) I'm speaking with Max Nadeau. Max works at Coefficient Giving, which is the philanthropic funder that was previously known as Open Philanthropy. And his team, the technical AI safety team, makes grants to support technical efforts to mitigate catastrophic risks from AI. So that's things like interpretability, alignment research, evaluations of AI behavior, and so on. I'm talking to Max today because Coefficient Giving is launching a new funding initiative, basically getting people to start non-profits that plug the biggest gaps in technical AI safety. And there's quite a lot of money on the table. So there's between $200,000 to $200 million of funding available for a new non-profit, depending, of course, on the... stage of the grant and how promising your proposal actually is. But I'm just super excited to dig into this. So Max, thank you so much for joining us. Speaker 1 (1:42) Thanks for having me. Speaker 2 (1:43) Right. And before we begin, I should disclose that Coefficient Giving is 80,000 Hours' largest donor, although we haven't received funding directly from Max's team. So yeah, you were previously at Redwood Research. I'm interested to hear what you got up to while you were there. What did you do? What's your origin story? Speaker 1 (2:07) Yeah. So the primary work I was doing at Redwood was actually field building rather than direct technical AI safety research. So I was running the machine learning for alignment boot camp that Redwood ran. And then I also was running the Redwood. Model Internals Experiment, which was shortened to Remix. And both of those were programs in which Redwood had something like 30 to 50 people come for a short period of time to learn more about AI safety research and develop their skills. Speaker 1 (2:46) And, you know, I'm happy to report that many of Speaker 2 (2:49) the Speaker 1 (2:49) people who went through those programs went on to full-time work in AI safety. I also dabbled a little bit at Redwood in doing interpretability work directly. But frankly, I didn't really like doing the technical AI safety research myself very much. And so... it was a better fit for me to do grant making work instead of being a researcher directly. Now, I think maybe now that the AI agents do all of the drudgery, I might enjoy technical AI safety research more these days if I were working on it. But I think I'm pretty happy in my current career trajectory too. Speaker 2 (3:23) Got it. So you're launching this new initiative called Project Tailwind. Can you tell us a bit about what this project is specifically? Speaker 1 (3:34) Yes, definitely. So Tailwind is a new initiative that we're starting at Coefficient Giving to get more philanthropic startups in the AI safety space. And these are primarily, but not exclusively, technical startups. We've got a long list of, I think, 40 or 45 project stubs on the Tailwind website, which you can read, which are all ideas for projects and organizations in the AI safety space that we've identified as being major opportunities for impact that we need someone to found. So we are making grants of a couple of different types. So we're making pre-seed grants to founders who may not have established. a team yet, may not have a super detailed vision for what the organization will do, but have some preliminary ideas. And those pre-seed grants are between $200K and $2 million. Then we're making seed grants, which we can do right out of the gate if the application has the important qualities. And those important qualities are there needs to be Some amount of a founding team, there need to be like multiple people who are covering the important bases for the organization. And some initial, obviously very preliminary results, either technical or otherwise, showing that they have the capacity to make progress on the goals that they have as an organization. And those seed grants are going to be between $2 million and $20 million. And then for some organizations, Speaker 1 (5:10) which this could be the first grant that we make them, but more likely it will be the second or third grant once they've had some time to establish themselves and set up a track record. We're willing to make grants up to $200 million, even for new organizations right out of the gate. And I think longer term, I mean, $200 million is not an upper limit on the largest size of a grant coefficient giving we'll ever give to an organization in the technical AI safety space. Speaker 1 (5:38) I personally would predict that at some point in the next year or two, we'll make grants that are higher than that. But obviously, that will depend on many factors like how good the organizations are and how much the funders want to support that and all sorts of other things like that. Speaker 2 (5:55) And in terms of your approach these days, you know, one way that you're kind of replicating the venture capital model, I understand, is that you're just like moving quite fast. Are there any other things that you're borrowing from the VC world? Speaker 1 (6:10) So one thing which has always been a part of coefficient giving's philosophy is the notion of hits-based giving, which is a borrowing of hits-based investing from the investment venture capital world. And that's the idea that you make a bunch of grants and the majority of them don't have any impact, but a couple of them are so great that they make the whole portfolio worthwhile. And that's something that we're leaning even harder into right now. in light of the rapid progress in AI. So we're trying to make sure that we're not falling into a common failure mode for grant making institutions, which is to be overly focused on scrutinizing budgets and making sure that every cent is going to good use and making sure that in addition to due diligence, we're also paying a lot of attention to ensuring that the best projects have all of the funding they need and that we're really paying attention to the upside that's available from moving fast and supporting things generously. There are some other more tactical strategies that we're borrowing from the venture capital world. So for example, my colleague Jake Mendel has run a couple of pitch days where he has interested founders come and spend 15 minutes describing their plan for a new AI safety philanthropic startup, and then gets them a decision very quickly, like in under a week about whether or not we're going to make them a pre-seed grant. And we're also using scouts to try and help us identify some of the best people in the world to run these new startups that we're supporting. Speaker 2 (7:28) Yeah, super interesting. I am wondering, though, whether you like lose out on anything by taking this sort of faster, more dynamic approach. Like, does it mean that you get to be a bit less diligent than you ordinarily would like to be with your grant making? Speaker 1 (7:44) Well, I mean, I think tradeoffs of that sort are unavoidable. We're trying to keep them to an acceptable level, in particular by hiring more grant makers so that we can get answers to as many grant applicants as possible, as rapidly as possible, without trading off against diligence. Speaker 2 (8:02) Got it. Something I know that you've done recently is you gave, I think it was $160 million grant to an organization called Resolution. That's obviously like, that's massive. Can you tell us a bit about what makes a grant application exciting enough and promising enough to earn that amount of money from you? Speaker 1 (8:30) Yeah, yeah, totally. With Resolution, the CEO, Jeffrey Irving, is really just one of the luminaries, the leading lights in technical AI safety. So he was the chief scientist at the UK AI Security Institute. And before that, he led teams doing AI safety work at Google DeepMind and at OpenAI. And when he came to us, the plan, in addition to being like a great founder, the plan he had on paper sounded right up our alley. He wanted to start a large, ambitious research center that was going to work on work directly on the problem of aligning superintelligence, which is where we think a lot of the existential risk from AI comes from and is a subject that we think is sorely neglected by the existing portfolio of research bets in the alignment ecosystem. And I mean, just to throw on even more positive qualities, part of his plan is to spend a lot of time and effort on using AI productively to accelerate his safety work, which is something that's really, really important to us because we think that there will be, and to some extent are, opportunities to, you know, make... five years of progress in one in AI safety, while that is also happening in AI capabilities research. And we really, really want to make sure that our grantees are taking advantage of those opportunities. And so Jeffrey's enthusiasm for that approach to the problems he's solving was also a really big plus in our books. Speaker 2 (9:56) Yeah, awesome. So obviously, Jeffrey Irving is a, as you say, quite an esteemed guy. Speaker 2 (10:05) For these kind of quite big money grants, are you looking for founding teams that have quite big names in them? Or does a regular person have a chance of getting that amount of money for a really exciting project idea? Speaker 1 (10:20) I think right out of the gate, you need to have some pretty hard to fake signals of being a really thoughtful and... you know, well-intentioned founder, uh, if you want to get the sort of money that, uh, that we were able to support Jeffrey with. Um, but I think that we are very enthusiastic about, um, taking new organizations that are demonstrating really great results and having a lot of impact and scaling them up to that, to that scale, uh, within a. you know, a relatively short period of time after they're founded. But that would be a case where we'd be making the grant primarily based on the results of the organization rather than on the profile of the founder. Speaker 2 (11:06) So as I understand it, there are quite a lot of overheads involved in starting a new organization. And, you know, other downsides like smaller or newer organizations may just get taken less seriously than more established ones or be less well-connected and overall have a harder time or take a longer time to have an impact. Now, if very risky advanced AI systems are indeed coming quite soon, then I guess my question is, wouldn't it be better for coefficient giving to just go all in, like totally all in on scaling up existing organizations and helping them expand their focus areas? Shouldn't that be what you're prioritizing? Speaker 1 (11:58) So yes, we are definitely doing that as well. And in fact, yeah, I'm very, very excited about the grants that we're making in that space. And I think that for a lot of people who are going to potentially be interested in founding new organizations, I think they should really, really consider going to work at some of the existing third-party nonprofits in the space for all the reasons you list. I mean, these organizations already exist. They already have credibility. They already have processes in place. They already have years of, in many cases, years of thinking about what their theory of impact is. And so if you're a founder, you should be thinking like, okay, maybe I do want to start. a new team within Redwood Research, within Meter, within Truthful AI, or within Epoch, or within one of these other third-party organizations. So yeah, totally agree. I think, however, there are a lot of projects that are not being done right now and that are maybe not a very good fit for these sorts of organizations. And so those projects should happen too. Like, you know, basically we're trying to do an all-of-the-above strategy. And so for... People who don't want to work in one of these existing organizations or who want to do a project that wouldn't be a very good fit for the existing startups in the space, that's what Tailwind is for. We want those people to be able to go their own way and have their own startup that is able to pursue their vision for an impactful direction within AI safety. Speaker 2 (13:24) So I'm pretty curious here about what you think the bottlenecks actually are to getting all of this kind of work done. What are the main reasons that the field isn't already in a position, given that it already has several AI safety organizations, why is it not already in a position to do all of this kind of work? Speaker 1 (13:47) Yeah, so I think the most important bottleneck, maybe the only bottleneck, is talent. There are a number of really exciting AI safety organizations, but at each one of them, there's a long list of projects that they would love to do, engagements with AGI companies and with governments that they would love to take on that they just do not have the staff to execute on, which is a big reason that we're doing Tailwind and why we're trying to scale other parts of the AI safety field. We just need more talented people to come work on these problems. Speaker 2 (14:14) So yeah, you're saying that... talent is the biggest bottleneck right now for AI safety. Yeah. Can you tell me more about what kinds of talent we're most missing right now, especially in terms of the skills and profiles needed in founders and founding teams? Speaker 1 (14:34) Yeah. So, okay. I mean, I think for the purposes of this conversation, I'll mostly talk about... the founders specifically, although I do want to just make clear to the audience that I think there are lots and lots of opportunities for people who are not interested in founding new organizations to have a lot of impact in the AI safety space and in the third-party AI safety space. But I think that founders are a really important bottleneck. When it comes to what we're looking for in founders and what we think is most important for a founder in the AI safety space, I think there's a lot of attributes which are shared with normal for-profit startup founding. So it's really valuable to have perseverance, to be willing to run through walls. It's really important to have energy for not just heads down technical work, but also managing employees, dealing with funders. You know, generally like communicating what your organization does to a wider audience. And then I think there's also some distinctive attributes to AI safety philanthropic startups, which are not as important for the normal for-profit world. And the biggest one is being really discerning. and careful about your theory of impact. So really thinking deeply about what your organization is doing, how it's supposed to make a difference on future risks from AI systems that haven't been created yet, and being willing to rethink