OpenRouter: from Seed to Stripe — with OpenRouter’s Alex Atallah & AMP’s Anjney Midha

Latent Space: The AI Engineer Podcast

This episode explores the evolution of OpenRouter from a simple marketplace for LLMs to a critical piece of infrastructure for the AI ecosyste

Key takeaways

• Alex Atallah applies a PubSub product principle where developers subscribe to model SKUs as continuous data streams.
• The release of Alpaca proved that high-quality models could be fine-tuned for incredibly low costs, democratizing the ecosystem.

Main topics

• OpenRouter origin story
• The rise of open-weight models

Notable quotes

Models are not products where you can just enumerate all their features onto a web page. They're all black boxes.

Conclusion

OpenRouter is positioning itself as the essential, secure routing laye

Transcript preview

Speaker 1 (0:02) Okay, we are here in Andres' house. It was just where all great startups in San Francisco start. Howdy. And congrats on Cursor, Mistral. God knows what else. You've got so much stuff going on. There's Speaker 2 (0:17) a lot going on. Well, OpenRouter Speaker 1 (0:19) has probably been Speaker 2 (0:19) the most, I would say, one I'm excited about recently. Yeah, Speaker 1 (0:24) and we have Alex. First time on the pod, but you've been there a few times. I appreciate every time you've shown up. for the community. Congrats. I just like, what a journey. When I was looking back at your past posts, one of the earliest principles that I saw you write as a sort of product person is PubSub as a product principle. And I wanted you to maybe explain how you think about what should exist in the world. Yeah, the Speaker 3 (0:49) PubSub piece, which was early 2023, I didn't think about it until we talked like 10 minutes ago, is about how There is like a way of thinking about products as an intersection between subscribing to data and publishing data. And marketplaces are an easy, easy, easy example of this. You have suppliers that are publishing some kind of product to a SKU. And the SKU is kind of like a PubSub topic that a consumer is subscribing to and just going to like consume whatever they want. And humans consume in a very like discreet, ad hoc way. It's not very scalable. You know, all their attention is on the topic when they're buying the thing and their attention is nowhere else when that happens. Agents and consumers of inference don't act like that. They're consuming continuously and they're changing the skews that they consume from all the time. So open router is sort of like a blend between a normal API experience and a marketplace where we create model slug, we have the auto router, we have all kinds of like product SKUs that you can subscribe to. And then you can like continuously add like derived value and make decisions based on those consumers. Speaker 1 (2:11) Yeah. This is something that was more consensus now, but not consensus when you guys started, which was... that there is such a demand for swapping models and changing things out and that people would not use the native SDKs. I guess for each of you, what was your realization moment that this would be it? You've given a talk at AIE about Alpaca as one of your inspiring moments. Speaker 3 (2:35) Alpaca, I can rehash the Alpaca moment for a sec. The very beginning, at the end of 2022, OpenAI was the only game in town. There was OpenAI. Speaker 3 (2:47) And then a smattering of early attempts at open weight models. When WAMA came out in January of 2023, it was like, whoa, really exciting. This is really big. It outperforms GPT-3 on one or two benchmarks. But you can't chat with it. It wasn't actually an engaging model. But it seemed like someone just needed to fix a couple of things and do some RLHF on it to get it all the way there. And Alpaca was the first model that I saw that did that. It only took $600 to do. A team at Stanford generated a bunch of synthetic data, fine-tuned LAMA, and made Alpaca, 7 billion parameter model. Maybe it was 13 billion parameters. And it was so good. Like, I was just... like on an airplane using it. I, you know, in many cases, like you could not discern a chat GPT versus an alpaca result. And I figured if it was this easy to make a model, one, we have a whole new way of monetizing data for the first time. You can just like take really valuable data and turn it into a service in $600. And that cost will probably go down over time. Speaker 1 (4:03) When you say monetizing your data as what eventually will become an MCP endpoint or as training data for a model? Speaker 3 (4:11) Yeah, training data for a model. Like an abstract way of saying like, hey, I have this data. Compress it into a model. Like it makes sense for me in my product, but like I could repackage it in the form of a model and sell it. And so it's just a whole new business model for the economy. It also, of course, provides like... you know, a way of following what frontier labs are doing, but in a way that like a single developer or a small team of developers can roll on their own. And so whenever you have an example of that, like a breakout app that's doing really well, and then some kind of framework for imitating it in your own flavor, you have an immediate ecosystem of like... An immediate ecosystem should arise because there's just a huge gap between the decisions that the single company is making and all of the variations in those decisions that a wider ecosystem can create themselves. And so then you need a marketplace to discover all of those services and all of those products. There wasn't any place on the internet. that was like a home base for LLMs in terms of seeing how much they were being used and seeing who was using them and why. Speaker 1 (5:29) The closest would be Speaker 3 (5:29) Hugging Face. Speaker 1 (5:30) Hugging Face Speaker 3 (5:30) was the closest. They just started Hub a few years ago, before that. Yeah, and Hugging Face also didn't have the closed-source models. Yes. And you couldn't use the models at the time. And there wasn't data about who was using them. There are a bunch of differences between OpenRouter and Hugging Face. Speaker 3 (5:50) those differences felt really critical to me, especially when I was just trying to learn about LLMs and why people are choosing these different little ones that are emerging over Speaker 1 (6:02) time. Got it. And then Ansh, no stranger to wanting more model diversity. At the time, you're a couple of years into your Anthropic journey, which we've covered in the previous podcast as well. What was your introduction to Alex? Well, the introduction was... Speaker 2 (6:18) I think 13 years before that. Oh. But the open router handshake actually happened right over there, if you remember. Yeah. Which was, Alex and I met, I believe it was sophomores now, if I remember correctly, at a Stanford Review meeting for the first time. I think so, yeah. Yeah. So Stanford Review was the libertarian newspaper on campus at Stanford that Peter Thiel started back in the day. And for whatever reason, Alex and I both showed up to one of the And I remember the editor-in-chief was a mutual friend of ours. Lisa was a really great editor-in-chief. Part of an editor-in-chief's job is to assign responsibilities to people and make sure the work gets done. And I may be misremembering the details, but I remember wanting to... It was kind of surprising to me that at the time there was no dedicated technology section in the newspaper. Speaker 2 (7:12) Because it's political, right? It's primarily, it originally started as like a political, yes. Yeah, Speaker 1 (7:17) states and those Speaker 2 (7:17) things, Speaker 1 (7:18) yeah. Speaker 2 (7:18) But to take us back in time, you may remember this, but there was this technology kind of legislation that was being debated called the Net Neutrality Act. And net neutrality is like inherently this political concept, right? It's about the regulation of internet broadband access. And so there was a community of us who were kind of... but also debating the politics of the technology. And I thought the review would be a great place to write about that. And I was working on, I think, a net neutrality article. And I remember proposing, well, maybe you should start a technology kind of section. And Alex was one of the only people who said, yes, that would be cool. And so I forget whether we ended up writing stuff together, but that's when we first met, was 2011 or 12. I Speaker 1 (8:06) forget which year it was. It was one of those. Yeah. It's at Speaker 2 (8:09) Old Union, Speaker 1 (8:10) if I remember correctly. That's where we used to meet. Speaker 2 (8:13) But, you know, along the way, Alex and I've had a chance to... to hang out often. And probably the time when we had the most professional overlap was when I was running the platform at Discord and it had become this explosive kind of platform for crypto and Speaker 1 (8:32) NFTs Speaker 2 (8:33) in the middle of the pandemic. Speaker 1 (8:35) Which also, by the way, you were in charge of safety and security as well, right? I was the head Speaker 2 (8:38) of platform, which meant all of the crypto, the DAO and NFT launch. security debugging fell on me. And the phishing. The phishing, the social engineering attacks, like a ton of DDoS that we were getting hit by. It's around the time I started teaching security at Stanford, CS153. And Alex was at OpenSea at the time, and I was trying to figure out how we could defend against all these attacks that we were... And at peak, I forget if you remember how much NFT volume was running through. Discord. Discord. But it was like a meaningful amount of like several billion dollars in NFT volume of GMV, so to speak, were running through the platform. And it was all coming from OpenSea. It was these like buy, sell, trade. I mean, the D in DAO is Discord. Speaker 1 (9:23) Yes. Speaker 2 (9:25) So that's when I think we had hung out professionally. But a year after that, OpenAI gave Discord early access to GPT. Sorry, GPT-3. No, it was GPT-3.5, actually. Yeah, GPT-3.5, which is the RL version of GPT-3. And that's around the time we made a Discord bot with OpenAI for internal deployment. And that's when I realized we would need, like, since I was part of the deployment team. What was the use case? There were two that were, and there's actually a post now called Discord is your place for AI with friends that somebody sent me recently that I wrote and published in 2023. There were two use cases. One was Clyde, which was like a first-party friend inside of Discord that could help you set up your Discord server and talk to you about onboarding and get your friends to hang out more. And then there was content moderation. And one of the realizations we had with content moderation was it would refuse to moderate. Like, it would just refuse our prompts because the RL... The post-training was, we were very early in the post-training era. And it would just, our prompts would trigger it. It's like guardrails. And we told OpenAI, hey guys, we need access to the weights because if we're going to be doing content moderation at scale, we have 250 million monthly active users. We need more reliability that the model will do what we need it to. And they said, well, sorry guys, that's not how this works. We're a closed source company. And so that was my first realization that we needed open models and the enterprises would need more. control over capabilities and then ultimately would need some kind of control plane or management system to orchestrate these open models. But there were no good open alternatives until maybe six months later when Lama came out. And six months after that, I led the series into Mistral, which was started by Guillaume and the Lama team. And around that time is when I remember hearing about Alex launching OpenRouter and going, These worlds are going to collide. And I don't know when it'll make sense to team up, but Alex was so early and could see, I think he was totally right about this ecosystem starting with Lama that then needed like an easy layer to manage for, especially for, I was approaching it from the enterprise perspective because I'd been that, like as a VP of platform at Discord, it was my job to ensure that when we deployed models to like 250 million users, they did what we wanted them to. And that was very hard because if you outsourced it to the labs and they controlled the guardrails and their guardrails or their safety policies forbid the model from responding to your prompts, that was quite catastrophic. Speaker 1 (12:05) Yeah, but moderation is a thing that they want to support. And obviously beyond that, OpenAI would work with you, presumably to give you a moderation endpoint, which they offer for free. Speaker 2 (12:16) It was an interesting use case that they, so they did give us a moderation endpoint. However, as you guys know, every Discord server is like a mini deployment of itself. And so the use case was instead of having human moderators that have to interpret the norms of the community, you just give the, they often, like every subreddit, Discord servers, public ones, have their own rules that the user, the user's created. Oh Speaker 1 (12:41) yeah, we Speaker 2 (12:41) run the LensBase Discord. And then humans used to, read those norms and then enforce it every day manually, like observing each message in these communities. And these communities have like millions of users. So we had a 5,000 plus person team globally on the Discord content moderation team. These are outsourced contractors who had a really tough job. And so the idea was instead, if you could give the norms of that server to the LLM, then the LLM would do custom moderation for that server. It's almost like in context. moderation for that server. And many of those servers norms just violated OpenAI's rules. And so that it was like we had our own custom eval. So each server had its own custom eval. But at the time OpenAI's evals, we were also primitive in our thinking about how to deploy these LLMs. And often the post-training prompts were super heavy handed. It said, oh, anything about Harry Potter, anything that has trademarked content, you know, don't refuse. And if it was a Harry Potter fan community, this is a real use case, that had a content moderation, the LLM would just refuse. And that was just not precise enough. Speaker 3 (13:51) Another one that we heard was like, if someone was trying to write like a detective story, and there's one chapter with a lot of violence, like maybe someone kills someone, the LLMs would just refuse to like help with that part of the story. And then like these would be like, okay. This is not structurally inherent to LLMs. There must be some choice out there so that I can switch to another model when I'm getting a refusal or a bad result from the main one that I have. And that tension also drove me for a marketplace. Speaker 1 (14:28) Yeah, I think that is well accepted now. What was it like back then when you were raising or starting this? Did people get it? you know, what was some of the struggles? Basically, I like getting stories out of him about how other VCs don't get it. So, like, anything you want to talk about, you know, now that, let's call it, you know, the early journey of OpenRouter is done. You can obviously talk about some of the early days stuff. Speaker 3 (14:54) Well, I was going to say that, like, the biggest objection we got is big model win, which is Speaker 2 (15:01) all the value. Speaker 3 (15:04) Scaling loss. Yeah, scaling laws and natural network effects are just going to kind of accrue to one company, which will be like, it'll be a Google-style monopoly, just like how Google won the search market by a large, large margin. And you'll just be fighting for scraps at the end, basically. That was probably the biggest objection we got. You know, it is interesting that Google won the search engine race with such a huge margin. You know, I think like had there been more interesting benchmarks or had like search engines been, you know, had people like seen them a little bit more like LLMs where there are services that you can build companies on top of, that might not have been the case. But LLMs don't merely have a user interface. They're also like ways of building entirely new businesses and, you know, a Google level monopoly. would be like the Dutch East India Company times a quadrillion in magnitude because the whole economy ends up depending on the one monopoly as well. So it didn't seem like, you know, like would be a really crazy outcome if that happened. And it's also less likely because the economics of like creating good competitors are much, like much more decentralizable. Speaker 2 (16:24) Everything Alex said is true. And I came, Added from a completely different perspective, which is... Yes, this is why we're here. The scaling laws were never like, in my mind, were always a feature, not a bug for why OpenRouter would be very valuable. Because I was one of the first investors in Anthropic. And it was obvious to me that other researchers in our friends group, I went to grad school for machine learning. And I just had a lot of friends in the ML community who, it was very obvious to us that the bitter lesson holds. And so I was like, oh, fantastic. Now we have at least two proof points that compute scaling works. It was OpenAI and Anthropic. And by the time I think we decided to team up on OpenRouter, I'd already invested in Mistral and Black Forest Labs and Luma. So there was Speaker 1 (17:09) multiple companies and teams that I was working with. But you did other modalities, whereas this is literally text. Speaker 2 (17:18) Exactly. And it was so obvious to me that an ecosystem of different kinds of models were being created and that this whole narrative of like only one company will dominate like Google was like maybe true, but one, I don't believe that. But two, there was so much extraordinary innovation happening across several different research teams. But the shared problem I was noticing across all of them was often, you know, the research teams were fantastic at figuring out how to reason about new capabilities. They think in terms of capabilities, but never, like are not developer mindset oriented. Like what happens after the training is done and the checkpoint comes out? Like you'd be shocked how like similar their early pre-training teams at OpenAI, sorry, Anthropic, BFL, Mistral, were in their like default approach to taking their research out of the lab. kind of scaling their impact, which is often, oh, the checkpoint is done, put it out as an API, done, and then there'd be crickets. In the case of Cloud, the first Cloud checkpoint was actually done a year before they released it internally. And then ChatGPT came out and we decided, okay, yes, it's a good idea to release a Cloud version externally. And they had no plan, like no plan for how to get developers to actually try it out. And so if you go to the Cloud... one blog post, you'll notice they like three kind of developer examples for users of the API. And one is a discord bot. And the second is Vivian, my wife's startup called Juni Learning. Cause, and then there was like Notion, um, because these were all friends of like the Anthropic team. Cause that's how hard, like last minute, the planning was around, Hey, once the model is done training, how do you get it out to the world? There was no distribution platform that understood what developers needed. All the, I, the. key management, provisioning, like simple, like endpoint management, versioning control, like all these things that the scientists and researchers go, I mean, that's plumbing. I don't really think Speaker 1 (19:15) about it. Speaker 2 (19:15) Implementation detail. Right. And instead, Alex came at it from that perspective. And so, you know, it was so obvious to me that like every single lab I was funding would spend like literally sometimes billions of dollars into training. And then a checkpoint would be done and there'd be crickets like during early access because they're like, oh, that's right. Like, it's hard to use a checkpoint to make anything. You actually need a whole bunch of plumbing around it to make it usable by a developer. And so by the time, I think it was so obvious to me that a distribution platform like OpenRouter was critical to having the ecosystem if we wanted there to be competition to Google. Like, unless, you know, with Google, DeepMind is done training a new checkpoint, and then they push a button and it gets blasted out across all their surfaces from Google Docs to... Everywhere, even if I don't want Speaker 1 (20:02) to. Speaker 2 (20:02) Everywhere. Like on Android, like overnight, they can deploy a new checkpoint to like a billion devices, right? And that invisible infra advantage, distribution advantage, most people don't realize, but until OpenRouter showed up, you had to think about all of that yourself as a model lab. And it was very daunting. At Anthropic, I think it took more than 12 months to get to our first 10 million in revenue. And in contrast with Black Forest Labs, I remember the early days, you... You guys had a conversation with the BFL team and it was so simple for OpenRouter to say, oh, no problem. The day you launch, we can send a million developers to you. That was crazy. That was like a step function change in like an hour. Is that a real number, a million? I think today it's like four million. How many developers are on OpenRouter today? Speaker 3 (20:52) Over 10 million, but it's hard to... Yeah, I don't know how Speaker 1 (20:59) to, Speaker 3 (21:00) we do a lot of like, you know, account de-duping work, but you Speaker 2 (21:03) know, no one. If you could get a thousand developers, just to