Frontier Chips for Frontier AI Labs, with Walter Goodwin, Founder/CEO of Fractile

No Priors: Artificial Intelligence | Technology | Startups

Walter Goodwin, founder and CEO of Fractile, joins Sarah Guo to discuss his full-stack AI chip company's strategic bets on high-bandwidth inference accelerators. He outlines how Fractile

Key takeaways

• Fractile operates as a ~150-person full-stack company spanning workload architecture, front-end design, physical implementation, and advanced packaging to maintain an agile development loop.
• The AI chip industry relies heavily on external ASIC houses like Broadcom for physical design and foundries like TSMC, creating structural bottlenecks that startups typically outsource.

Main topics

• Full-stack AI chip architecture
• Memory bandwidth vs. capacity tradeoffs

Notable quotes

"The snappier chatbot is kind of the faster horses of kind of fast inference."

Conclusion

The episode provides a grounded look at how specialized AI chip startups can compete with

Transcript preview

Speaker 2 (0:00) Right now we're trying to build a single chip. If you crack open an Nvidia system, it has anywhere between kind of six and nine custom chips all built by Nvidia to come together to build something that is really, really potent. So there's already a gap there. You know, it'd be great to be productive enough that we could start to build our own responses in that same kind of way. This idea of being able to have at any given moment a kind of rolling frontier of bets that you hope are deeply aligned, in which you're ready to trigger the ramp of. This is a huge advantage if you can kind of build that machine. and build that engine. It's the equivalent of the Frontier model for the chip space, as if you can just find a way to structurally carve out a three to six months advantage, you will be winning all of those deployments. Speaker 1 (0:46) Hi, listeners. Welcome back to No Priors. Today, I'm here with Walter Goodwin, founder and CEO of full-stack AI chip company Fractile. We talk about what it means to be full-stack as a company, the technical bets they're making, why he's so focused on memory bandwidth, his predictions for frontier model architectures in the future, and the structure of the chip market at the frontier today and tomorrow amongst NVIDIA, AMD, internal efforts, and this new class of accelerators. Welcome, Walter. Walter, thanks so much for doing this. Speaker 2 (1:17) Very good to be here, Sarah. Speaker 1 (1:19) You started this incredibly interesting company called Fractal. Can you give us an overview of what you guys do? Speaker 2 (1:23) Fractal is a chip company. We build very, very fast inference chips for the world's largest models. This sort of bet on speed is something which we've had right from the outset. started the company in summer of 2022. And I think at the time we were starting to see two things. One was obviously the arrival of foundation models trained on the internet and generalizing across everything. And the other was a few kind of wise people saying, we need to find a way to... pour more compute into these models at test time. We need to find a way to take what we had for kind of AlphaGo, where you have a capable neural network, but then it becomes superhuman as you roll it out at scale and bring that to kind of language modeling, for example. And so our... big pursuit over the past four years has been to find a way to build chips that will allow for kind of simultaneously taking these huge models and running them much, much, much faster than the chips of today, but do that in a way that kind of scales. So scales to models beyond the frontier today, scales to extraordinarily long contexts. And so that's the mission that we've been on. Speaker 1 (2:29) It's become a much more interesting chip landscape in the four years since you started the company. Where would you place yourself in the overall GPU and accelerator landscape for inference? Speaker 2 (2:41) Yeah, I'd say there's a kind of extraordinary zoo now of options available. And if you look across this kind of entire space of kind of AI ASICs, one of the things that I think is very striking is there's a lot of relatively identikit chips out there. And this is kind of a structural, it's Speaker 1 (3:00) a Speaker 2 (3:00) structural industry property where if you look especially across kind of hyperscaler efforts and Google kind of kicked this off with the TPU more than 10 years ago, you see this paradigm where although there are a number of kind of proprietary internal chips, Google with the TPU, Meta with MTIA, Microsoft with Maya, OpenAI now with Jalapeno. These chips that are ultimately developed and delivered in partnership with kind of a relatively small number of what we might call ASIC houses, these kind of back-end delivery houses, Broadcom is the largest, you know, $2 trillion company, that help people kind of realize these designs. And so when you kind of zoom into this sort of seeming embarrassment of riches in the spectrum of what exists today... you see that actually there's a lot of similarity across these platforms. All of these chips, you have HBM, this type of DRAM memory that is common to NVIDIA GPUs, AMD GPUs, and all of these other ASICs. You have the same kinds of bets on Tensor cores to do your matrix multiplications, the same advanced packaging with TSMC. And so I think one of the things that we see in the landscape is... there's a relative dearth still of kind of efforts that go. all the way up and down the silicon stack and try and build kind of fundamentally new capabilities. And this is kind of a structural thing. There's just not that many teams that have decided we're going to build from, you know, what you might call the kind of, you know, the architecture layer, the front end design layer, which mostly looks like kind of writing code, but also all the way down to what we call kind of physical design, the process technology, the kind of foundry interactions, the stuff that essentially has you flying back and forth to Taiwan or... every week, that's something which actually tends to still be left to a relatively small number of companies. For Speaker 1 (4:56) people who don't work in the chip industry, what is the common abstraction or handoff between what architecture-focused players are doing and a player like Broadcom? Speaker 2 (5:06) Yeah, so I think if you're, let's say you're Google and you're designing the TPU. you have a number of people in your team who have a really strong understanding of the workloads that they're trying to accelerate. And so, you know, going back over 10 years, this is a world that then sees you conceive of things like the tensor core, which is a designated circuit dreamt up. by an architect that is exceptionally good at matrix multiplications because you have this insight that matrix multiplications are the overwhelming majority of the kind of flop count in these models that you're running. Speaker 2 (5:43) That's kind of the preserve of the architect, I guess, is somebody who really understands the way that chips behave, but also ideally has a very strong understanding of the kind of target workloads. What you'll then see inside those same organizations is a set of sort of, you know, very smart people that kind of lower that down into, I guess, you know, essentially a kind of circuit level description of how this chip should behave. And so a lot of this is what we call front-end design. Speaker 2 (6:10) mechanistically it still looks like writing code on a computer and it's still code that is then handed off to a player like Broadcom. And so that kind of front-end design that describes the underlying intent of the chip, the underlying logic of the chip... is then transformed ultimately into something that, you know, what gets shipped to TSMC in the end is literally a kind of bitmap. You know, we have this file type you call GDS2, and it's literally, you know, where do I put the methyl layers? Where do I put each individual transistor? And so it's a full layout. And so you go through this process of synthesis of that RTL into a kind of set of circuits that then ultimately get laid out. And a lot of the complexity there is around... In Broadcom's case, it would be ownership of analog IP that is used for chip-to-chip connectivity. And it is this kind of physical placement that is specific to a given process node at TSMC. So we talk about like three nanometers, five nanometers, et cetera. So that piece is still today, generally it is a kind of outsourced activity for all of these projects. Speaker 1 (7:18) You are... describing what feels like a very large project to do end-to-end new chip creation. From a workload perspective, you're working with several frontier players to sort of characterize that and make sure that you can serve them. How did you think about, like, what does the Fractal team look like today? How can you do that as a startup? Speaker 2 (7:36) Yeah, I think there is a, you know, it's funny, there's a lot of industries where there will be a kind of waterfall style approach to doing things. And then suddenly, you know, along comes a new way of thinking. And it's kind of more, agile to borrow from like management philosophy right and i think For us as a very full stack company, so we have a team that does span, you know, very deep workload understanding. I think we actually try to, you know, run ahead in many places and look at, you know, where would we change a model architecture to be more exceptionally aligned with our bet? What do we think is like kind of the scaling law vector for the particular bets that we are taking? And then we can advocate to our customers and our partners on that. And it also helps us inform a set of bets for a chip that, you know, isn't going to necessarily... exist in volume for one or two years, that becomes a really important part of the muscle that we have to build. But then as we look across this ability to spin a very agile loop, it means that inside Fractal, we do have front-end designers, we have our own physical design team, we have our own back-end implementation team, we do our own work on advanced packaging, for example. And that's not an enormous headcount, right? I mean, Fractal today is about 150 people. We're