Inside Instacart's AI-Powered Smart Shopping Cart | NVIDIA AI Podcast Ep. 302

NVIDIA AI Podcast

David McIntosh, Chief Connected Stores Officer at Instacart, discusses the transformation of grocery shopping through 'Connected Stores.' He

Key takeaways

• The Caper Cart uses NVIDIA Jetson boards to perform real-time edge processing for weight and visual sensors.
• Instacart's sensor fusion system combines cameras, weight sensors, and SLAM location data to track basket contents.

Main topics

• Edge AI in retail
• Caper Smart Cart technology

Notable quotes

In five to ten years, customers shouldn't have to think about shopping in-store or online.

Conclusion

The episode demonstrates how edge AI and sensor fusion are turning phy

Transcript preview

Speaker 1 (0:00) Our view is that in five to ten years, customers shouldn't have to think about shopping in-store or online. There will be one single unified mode powered by this continuously learning AI system that incorporates what customers are doing in-store, online, states of the shelf to build a fully personalized experience. Speaker 2 (0:20) Welcome to the NVIDIA AI Podcast. Our guest today is David McIntosh. David is the Chief Connected Stores Officer at Instacart, and we're here to talk about the present and future of grocery shopping and AI in retail. David, welcome to the podcast. Thanks so much for joining us. Thank you for having me. So maybe we can start with the basics. You can tell us a little bit about yourself, your role at Instacart, and kind of your journey that brought you here. Speaker 1 (0:46) Yeah, happy to. So fundamentally, I'm a technology entrepreneur. My prior company, co-founder, CEO of Tenor, which is a expression search company, an animated GIF search company. If you're an animated GIF person, you probably use the product that is embedded in all the major messengers, keyboard companies, and so forth. When Google bought the company, we had a couple hundred million users, several hundred million queries per day. And then over three years at Google, we grew it to a billion users, over a billion queries a day. And what attracted me to Instacart was that I saw a company that is the leader in delivery online, but had an even broader market opportunity to really digitize the grocery industry, to bring technology to all of our grocery partners. And so in my first year at Instacart, I led what's called our enterprise business, the Instacart platform, and launched that. And you probably know Instacart as an app on your phone, right? A marketplace. But what's less well known about the company is we have a very significant enterprise business. So for example, if you go to Sprouts.com in the US, that entire experience, website, fulfillment. Ads all powered by by instacart. Yeah, and so as a result I would talk to retailers very frequently and what I heard was retailers saying hey instacart You brought me online you brought my business online website You know ecom loyalty etc, but I have all these problems in store I'm dealing with how do I get more of my customers to sign up for loyalty in store? How should I think about retail media in the store? How do I create a more personalized experience? And then on the other side, we had a lot of customers, a lot of users saying, Instacart, I love the convenience and delivery of the online experience, but I also like going to the store. I'm an omnichannel customer. How can you take what I love about the online experience, the convenience, the personalization, and we're getting to the store. The birth of Connected Store really sat at the intersection of those two insights. And so really our vision for the store is to digitize it end to end so that it becomes unified. Our view is that in five to ten years, customers shouldn't have to think about shopping in-store or online. It will be one single unified mode powered by this continuously learning AI system that incorporates what customers are doing in-store, online, states of the shelf to build a fully personalized experience. Speaker 2 (3:10) So you teased at this a little bit, but maybe we can kind of double click down. What does the term connected store mean at Instacart? And kind of along with that, maybe you can talk a little bit about how AI is being used, you know, as you said, Instacart, not just the grocery shopping app, but kind of across the enterprise services that you offer. You Speaker 1 (3:31) think about connected store is really digitizing the offline store today. So much of a grocery store today. is run very similar to the way it was run decades ago. If you think about the checkout experience, it hasn't fundamentally changed or evolved all that much. But technology has fundamentally shifted in that period of time. And so it's everything from our AI-powered smart cards, Keeper cards, which I'll talk about in a minute, to digitizing the deli, putting a screen behind the deli so that it's both easier for... associates at the store to prepare an order but also easier for customers to order it's connecting to electronic shelf labels so that it's easier for e-comm shoppers people picking orders to light up the shelf tag to more easily find what's on shelf it's about giving users an in-store mode so they can plan their trip to the store create a shopping list and then sync that with technologies in the store. It's inventory technologies that build an understanding of the shelf to prevent out of stock so that consumers, when they come to the store, could get exactly what they want. So you can think about it as really digitizing all the components of the store and connecting them together. So for example, you can order from the deli, from the caper cart. Caper cart connects to our offering called FoodStorm, which digitizes the deli. A shopper going into the store can activate those shelf tags, but you could also imagine the caper cart activating the shelf tag to help you find something more easily in the store. The shopping list that you build online, you can then sync with the caper cart in the store and it reminds you, in fact, what to get in the store so you don't miss anything and realize it when you're all the way home. So all these technologies are connected together. And at the very center of that ecosystem is caper, which is a $350 million acquisition I led a number of years ago. And the caper cart you can think about as a set of sensors in a cart connected with an NVIDIA Jetson board in every single cart. So the cart has a weights and measures certified scale. You need weights and measures certification in the US to do produce weighing. It's got camera sensors, multiple camera sensors that not only look at the basket, but also face the shelf. So you can understand what's on the shelf, which we'll talk about more. You've got... location sensors on the cart so you can understand where the cart is in the store it's both a slam approach but also leverages visuals okay Speaker 2 (5:55) of Speaker 1 (5:55) what's on the shelf and then it's a sensor fusion system so you can imagine in a grocery environment one of the big problems is that often wi-fi is spotty you know it drops in and out and often by the way there isn't good cell cell reception if you're going through you know these yeah a lot of the stores once Speaker 2 (6:12) you get in yeah at least i found the cell drops out it's Speaker 1 (6:15) it's it's a big problem and and when you think about it customers want an experience that responds immediately sure right so if you think about the way people use the cart they're putting items in the cart and then it creates this running total right so what people love about the cart is the fact that they can keep track of their their their spend right they don't have to go to the checkout line and then put things back because they miscalculated how much they're going to spend Speaker 2 (6:38) i'm laughing because i'm thinking as you're describing this, I'm thinking of one of your, one of Instacart's videos or ads that I saw where, you know, it's mom and daughter shopping and mom puts something in and it rings up and daughter puts something in and mom grabs it and takes it out. And you know, it shows how the card automatically deducts it from your total, which is amazing. And then daughter puts back in and mom's like, yeah, okay. But Speaker 1 (7:00) yeah, so maybe you can, we can kind of walk through what Speaker 2 (7:04) happens kind of behind the scenes. when I'm shopping and I want to set this up because I think this part just sounds so cool. It's these little simple things, right? But they really elevate the experience. And as a consumer, when you get a lot of little things together, you're like, oh, this is a great experience. But the idea that I can be shopping, I can have a bunch of stuff in my caper cart. It's keeping track of everything. And I can put something in a piece of produce that needs to be weighed. And so if I already have stuff in the cart and I grab an apple and I put it in the cart. what happens yeah Speaker 1 (7:36) great question so the entire basket is a scale and so when you put an item into the cart it can understand the weight because it already knows the cumulative weight of everything in the cart but as you're alluding to it's it's not that simple because you can imagine that the grocery environment is very complex right and so wi-fi as we talked about is one of the issues but look in these stores there's a variety of lighting conditions there's tens of thousands of skews the skews by the way change store to store. So even if it's the same banner, right, even if it's the same retail store, it's going to be very different catalogs for different items. The items change, Speaker 2 (8:15) right? Speaker 1 (8:16) Seasonal differences and so forth. There's very subtle difference in sizes of these items. And then the way that people shop is different, right? So going back to the question on the scale, some people are leaning on the side of the cart, right? Different arms are going in and out. It could be going over bumps in the store. And so it's a very complicated problem. the expectation from a consumer standpoint is that the responsiveness has to be in hundreds of milliseconds, right? And so if you think about a lot of the AI systems that exist in the cloud, the response time might be seconds, right? And so key to our approach is doing a lot of these calculations around what's in the basket and recommendations at the edge. And so the way it works is that we have several sensors coming together. We have camera sensors, we have the weight sensor. We have a location sensor and then we have that sensor fusion system. So we have an edge encoder that processes a lot of those signals at the edge. We also, for longer sessions, longer analysis, we will look at those sessions in the cloud with a separate decoder. Speaker 2 (9:17) And Speaker 1 (9:18) then we'll put the two together in what we call an overall shopping decoder, that shopping experience decoder, that builds the best understanding of the customer's basket. And you can imagine that in this environment, there's thousands and thousands of edge cases that can emerge in terms of the way that people are shopping with this product. And so we've really found that it's important to have multiple signals all coming together. In fact, we published a blog post on this, announced it at GTC, and actually walked through this in a GTC talk.