Open models and the future of Physical AI with NVIDIA
Practical AI
Daniel Whitenack and Chris Benson host Ming-Yu Liu, Vice President of Cosmos Lab at NVIDIA, to discuss the strategic shift toward open models and their critical role in advancing physical AI. The conversation
Key takeaways
- Ming-Yu Liu explains that open models are essential for innovation because they allow developers to inspect, fine-tune, and deploy AI locally without relying solely on restrictive APIs.
- Physical AI requires real-time inference and high efficiency per watt, making cloud-based API aggregation impractical for embedded devices like robots and self-driving cars.
Main topics
- Open models and ecosystem innovation
- Physical AI hardware constraints and inference
Notable quotes
"API are great. They solve problems, but they don't give you the insight."
Conclusion
The episode provides a clear roadmap for developers looking to transit
Transcript preview
Speaker 2 (0:01) Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show. Speaker 3 (0:41) Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at PredictionGuard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI and autonomy research engineer. How are Speaker 4 (0:55) you Speaker 3 (0:55) doing, Chris? Hey, I'm doing great today, Speaker 4 (0:57) Daniel. Really excited about today's conversation. Speaker 3 (1:00) Yeah, well, I think in the world that both of us inhabit, we are open to interesting discussions. I think we'll talk about things open and things related to the world in today's episode because we have with us Mingyu Liu, who is vice president of Cosmos Lab at NVIDIA. Welcome to the show, Mingyu. Speaker 1 (1:25) Hello, sent to have me here. Great to be here. Hi, Daniel and Chris. Speaker 3 (1:29) Yeah, yeah, great to have you. And one of the things I kind of alluded to is maybe something open, which I know is... dear to NVIDIA's heart, which is open models. And I've seen in the news recently, in what NVIDIA has posted, obviously they are promoting open models as a key piece of what is important in the AI ecosystem. I'm wondering, Mingyu, if you could help us understand, maybe from your perspective, from NVIDIA's perspective, why in today's world is open... is the idea of having open models and promoting open models. Why is that such a critical thing in the ecosystem? Speaker 1 (2:15) Yes. Speaker 3 (2:16) Okay. Speaker 1 (2:16) So when I was a student, I learned a lot about computer vision and machine learning. At that time, there were great researchers, students, open source on their code and model. And looking at their code and model actually helped me understand the concept and helped me to innovate. And the academy always worked like that and has been years, even before GPT. And GPT also had open source model in the past. And that had been pushed the field forward around innovation. It's just until recently. the model is getting bigger, the amount of resources required to build a model is tremendous amount, and hence, the amount of open model reduced. But open model generally is there for the field, and it's been very useful to push the field forward. And so, Amelia, while this continues to happen, I think the world would be in a better place if we have an open model to enable innovation. So people, you know, API are great. They solve problems, but they don't give you the insight. You know, it doesn't tell you, it doesn't allow you to tear apart and miss your idea inside, you know. And also, the kind of thing you want to do, maybe suitable. You want to have a local computer and you don't have an internet connection. So API may not always be the solution. And that's why I think an open model is important to enable people. NVIDIA is a serrated computing company. So we built this computer, a great computer, a GPU. networking devices, and now we also build CPU SoC. We build this computer and we want to enable the whole ecosystem. And we generally believe that to better build these computers, we need to know more about the applications. And in the old days, you have this computer and you have library and you have software applications. And in the modern world, You have computer and you have library still, but you now have models. Model is a new kind of libraries people can use to build great, amazing applications. You know, maybe all kinds of agents to help you buy stuff or help you to code, all on top of models. And we, NVIDIA, like to build models, like to build open models on top of NVIDIA's computer. There are a couple of reasons why we do that. First, through building the model, we better understand what kind of computer we should build, what kind of GPU architecture we should have, so that NVIDIA can always be there to help the whole ecosystem. And second, we also want to make sure people are enabled to do amazing applications. So when we have an open model there, people can test this out first and combine them to do applications that are good for their targeted business and gain some familiarity and maybe later they can build their own model. So there's a lot of reasons why an open model is critical both for the academy and also for the whole industry. Speaker 3 (5:55) Yeah, you were kind of getting to a follow-up that I wanted to have, which is it's important to have open models, to your point, to promote innovation. But I'm wondering, in what you're seeing at NVIDIA, is it increasingly important also on the commercial side? So like for NVIDIA's customers, right? What is the implication of... having access to open models maybe on the commercial side. Do you have any thoughts on that? So it's great if like I'm in a lab and I have five years to work towards the next innovation, the next types of models. But if I'm in a commercial company, maybe I'm a customer of NVIDIA, I run NVIDIA hardware. How does the existence of open models? help me as, you know, in the Speaker 1 (6:49) commercial world, maybe. Innovation, freedom. Innovation, freedom. I think that is maybe the two words to summarize. So, you know, you can leverage API to solve the problem. But with the open model, you can do more than what's available in API. So, I think it be the feel. is finding more and more important application of AI, right? You know, from earlier day of chatbot and now the coding agent and we also see more and more agent that can do things that we didn't think it's possible. People are working very hard to try to figure out how AI can make our life collectively better, right? And there will be application that was not foreseen by... the foundation model of the frontier companies who are building AI, right? And they require some customization. Open model allow them, give them options, give them a choice to do the innovation they need for their target business. And this way, I think it's good, you know, for the whole society. Speaker 4 (8:04) I think, I know I was, you know, kind of following up on your point there. I was very happy recently to see NVIDIA kind of, you know, there was a moment some months back where, you know, open models in the West were starting to struggle a bit, you know, and some certain unnamed organizations were closing and that had been leading the way. And so it was a delight to see NVIDIA step in and take the leadership there and follow through with some other things, the hugging face acquisition that's coming in. the work that you're doing. So we're kind of going through this turning point toward the next generation of open models and what that looks like, especially with regard to the landscape of physical AI out there, which is my personal passion, as most of our viewers know. And so I would love if you could... kind of, as we are pivoting into this new generation of open models in a physical AI context, where it's not all API-based, it's not all frontier models at big services and stuff like that. And we're all getting excited about drones in everyday life, robots in everyday life, and all the different capacities that that can... can bring benefits. Could you kind of lay out the landscape? What is this, as you mesh these things together, what does that look like to you now? You know, what are you seeing as the opportunities in the near term? And kind of how are organizations that work with NVIDIA starting to position to take on this next generation of physical AI that's upon us? Speaker 1 (9:55) Yeah, thanks for the great question. So I think physical AI, is a term that covers several important verticals. Automotive is one. Factory automation is another. And robotics is a third pillar. There could be more. Agriculture, construction, many of them. And physical AI are AI deployed in physical device. And physical device perturbs the state of the world and completes certain tasks. And the result is shown as material, as the final something you can touch. That's how I would like to define physical AI. So nowadays, we have some AI in physical device, but they are preliminary. And I think the most exciting form of physical AI, maybe one of the most exciting forms of physical AI is a humanoid robot. And we envision a humanoid robot will be able to help people accomplish a lot of tasks. And now we don't have a humanoid robot surrounding us, but one day we have a humanoid robot surrounding us. they will require a computer. They will require a powerful computer to help us process visual input, understand people's instruction, and then self-correction in company tasks. And these are great business opportunities for NVIDIA. So that's why NVIDIA cares about physical AI. If we can make physical AI come, the dream of physical AI come true, there will be a lot of opportunities for NVIDIA. So we are very aligned. So we want to help the field moving forward. Speaker 3 (11:39) Yeah, go ahead. Yeah, and I was wondering, just tying that back to where we started the