77: Automate All the Things!? A UX Reality Check on AI (ft. Dr Stefanie Hutka)

Finding Our Way

Dr. Stefanie Hutka joins the hosts to discuss how historical lessons from industrial automation can guide modern AI transformation. The conv

Key takeaways

• AI functions as an 'accelerant' or 'amplifier' that scales either positive innovation or existing organizational dysfunction.
• The concept of '10x cognitive offloading' via generative AI risks de-skilling humans if the entire Observe, Orient, Decision, Act loop is delegated to agents.

Main topics

• History of industrial automation
• AI as an organizational amplifier

Notable quotes

AI is an accelerant. What are you feeding it?

Conclusion

The episode serves as a strategic guide for leaders to navigate AI by

Transcript preview

Speaker 2 (0:00) Hey everybody, before we jump into the show, I wanted to share with you something that Peter and I have been working on. It's kind of special. We have poured all of the thinking and all of the wisdom that's come out of the last several years of this show into a six-week leadership skills development and support program that we call the Intentional Design Leadership Circle. It's happening October 7th through November 10th, and there are only 20 seats for this live online program facilitated by Peter and myself. Find out more information and get your receipt at findingourway.design. Now on with the show. I'm Jesse James Gehrig. Speaker 3 (0:43) And I'm Peter Merholtz. And we're Finding Our Way. Navigating the opportunities and challenges of design and design leadership. Speaker 2 (0:58) On today's show, we're joined by Dr. Stephanie Hutka, design professor, researcher, and author of the book, What Your Machines Should Do, The Science and Strategy of Human-Centered Automation. She'll share lessons from the history of industrial automation to observe models to follow and models to avoid as organizations move towards AI transformation. From the humble loom to the factory floor to the values that underpin our processes, let's go exploring. with Stephanie Hutka. Stephanie, thank you so much for joining us. Glad Speaker 1 (1:31) to be here, Peter and Jesse. Speaker 3 (1:32) To kick off, I think it would be good just to get a sense of how you introduce yourself. I know you teach and you write and you speak and you work. And so who are you? Who is Stephanie Hutka in the world? Speaker 1 (1:42) Yeah, so I've been introducing myself by way of hats, that I've been wearing a couple of different hats. Hat number one, and arguably the largest hat, the top hat, if you will, is as a researcher. So I have even more sub-hats under that field. I go back to early days, my PhDs in cognitive neuroscience as very much an academic researcher. I worked as a research scientist, found my way into design research, and that is primarily the type of work I do today at my consultancy called Sendful that I've been running for about three or so years now. So hat number one. researcher, specifically design researcher today. Then hat number two is as educator. So I've been teaching at UC Berkeley, primarily at the School of Information, but also at Jacobs, the design school. I teach introduction to user experience design. I just started my fifth year of teaching that to professional master's students. I also have designed some new courses, including UX for AI and designing future systems, which dove into some systems thinking, mindsets and toolkits, as well as some strategic foresight. And then the third hat is as author. So I've written various things over the years, scientific articles, industry articles. Most recently, I have a new book that is coming out, my first book, What Your Machines Should Do, The Science and Strategy of Human-Centered Automation with Rosenfeld Media that will be launching in November and currently on pre-order. Speaker 3 (3:12) Wow. Jesse and I have been exploring the subject of AI and design, primarily design leadership for the last almost two years now. And when we found out about what you were writing about, we're intrigued. And so unpack a little bit what the book is about for an audience of design and user experience leaders who maybe are still kind of grappling with concepts of automation. For Speaker 1 (3:38) sure. So the book is a field guide for product and design decision makers to navigate this question of how do we automate and closing what I've called the automation strategy gap, where you have these dreams of fully automated futures and running after let's automate everything versus the reality of what's actually valuable to our users. How does this fit within their ecosystem? What is our organizational readiness? What is the actual capability of the tools at this time compared to that fully automated future? And so the book shares different frameworks and tools, some of which are derived from my own practice, some of which are derived from a synthesis of literature that I like to say is intended to be durable. It's evergreen stuff. This is not a book about how to use cloud design or whatever tool is going to ship by the time this podcast finishes recording. I'm really looking at... What can we learn across human factors, human computer interaction, management theory, cognitive science, and acknowledge that we are in a very new territory. There's many ways in which generative AI is unprecedented. Those three ways that I usually refer to that is the unprecedented pace of development, the breadth of use cases that it can cover and what that means for human cognition and how much we can offload, and then the stochasticity of the output. So it is definitely, you know, it has unprecedented elements. That said, we have been automating stuff. We have been building autonomous systems for quite some time. So what can we learn from those principles so it doesn't feel like we're completely flailing as if there's no prior, prior is plural. So the book is about those things. Speaker 2 (5:17) One of the interesting debates about this current wave of AI technology recently is the question of well, how radical a change is this? And there is definitely this school of thought that holds that AI is quote unquote normal technology. It's just another wave of technological change. It's not very different from the things that have gone before. And then there's this opposing camp, which is very much on the side that this is so fundamentally different that there's not a lot that we can learn from the past. And I wonder where you fall on that continuum. Speaker 1 (5:55) Yes, a little bit of column A or column B, or for my fellow design researchers listening, it depends. Maybe how are we defining AI? Assuming we're talking about generative AI, because there are many types of AI. But yes, like the AI moment that we are in. I'll start with maybe column B of what might be similar. So there's something from one of your past episodes. I took notes on this because I was excited about this Speaker 2 (6:18) conversation. She's a researcher. Speaker 1 (6:20) Exactly. I was doing research, secondary research. This is what I do. The Paul Ford episode. from a couple months back, you talk about compilers as an example of now you have this sort of translation layer of intent going into the ones and zeros. And we've had moments like this where we can articulate our intent in different ways that speed up our processes. So from a technological standpoint, there have been big advances. We've had disruptive innovation. We've had mobile. We have had cloud. And those are definitely major events we've had. various tech bubbles before we have navigated those. So we have, I think, from a disruptive innovation standpoint, we have some priors in terms of designing automated systems, like moving from human in the loop to human on the loop. I mean, there are planes flying over us right now with autopilot systems that were presumably pretty comfortable. We're not too worried about how will the pilot lock back in with situation awareness when they need to. And we have figured some of these things out. So maybe we can learn from some of that. So that's what I'm talking about with some of those priors. Now, where I think it is different right now is what I've called 10x cognitive offloading. And what I mean by that is cognitive offloading is this just phenomena of delegating our thinking tasks to some sort of external device and often technology. The classic example is Google Maps. I give this example, I bike to campus to South Hall. I've been doing that route for five years. I've got that locked in. But the moment someone invites me, say to cheese board, which if you live in the East Bay, both of you know this, it's not far. It's like a mile from South Hall, but I don't do that trip very often. And I am entirely reliant on Google Maps. So I've offloaded my navigation capabilities to this digital map. And typically, when we've offloaded to technology, it's been relatively finite in terms of our tasks. And the way, at least by default, a lot of these generative AI systems have been designed, like the phenomenon of the prop bar, it really invites kind of wholesale offloading. And we're delegating the entire Observe, Orient, Decision, Act loop increasingly with agents such that if we are offloading quote unquote the wrong thing, we can very quickly start to de-skill. If we haven't learned the thing in the first place, there's going to be a gap in our learning. And I think getting to those three things I mentioned at the top of the podcast around one of those being the breadth of use cases to which you can apply this technology, I think that is where it is different. And combined with the pace and the inability for slower moving systems, governance layer, for example, we can't build guardrails as fast as the tech is moving. I think that's where things get unique. Speaker 2 (9:04) It's interesting to think about that pace challenge for organizations as they are trying to adapt themselves and adopt these technologies and actually get some meaningful value out of them in that. the technology is changing so fast that organizations hardly have the chance to orient toward one approach before the state of the art changes so if we put this in the context of a larger history of industrial automation right where we've had increasingly more and more machines doing things that humans used to do if we can define automation that broadly. We are now in a place where there is this kind of this shearing force where the technology is moving so much faster than the organization that the organization fundamentally breaks because of it. And I wonder about in your research, from your perspective, what we can learn from the pressures that have been applied to organizations in the past from past waves of innovation that might Teach us something about what's going to make an organization resilient now in the face of what's happening. Speaker 1 (10:18) Sure. So what? Immediately comes to mind is a contrast that I draw in a chapter where I get into systems mapping, systems thinking, is contrasting Toyota production system with General Motors approach. So Toyota production system, one of the key tenets of this is automation with a human touch. Jidoka comes from the Toyota founder. He was a young boy, saw his mother working at a loom and saw that she constantly had to be monitoring this loom and invented this mechanism such that when the thread ran out or got stuck, the loom knew when to stop. And this was ultimately like fast forward, Toyota moves from textiles into auto manufacturing, Japan devastated after World War II, American automakers, very, very resource rich. And you get to Toyota saying, okay, how do we move forward given our limited resources? And you have this approach to automation. It's very complimentary to people. You have this idea of kind of the and-on cord. You pull it if there's an issue on the line and everyone comes rushing. And I think that's interesting because it speaks to kind of the values that the company sort of organized around. Certainly there's contextual factors. There's different geographical region. cultural value, so on and so forth. And there's a psychological safety when you're pulling that cord that you feel like, okay, I can pull that cord. So that's an example of maybe automation done arguably well versus a much more sort of Taylorist approach that I think we see with General Motors as a contrast, but we also see in a lot of our thinking now. So in contrast, you have this mindset from like the early 1900s, if we can atomize work. And we can easily swap someone out to do it. And I think there's an underlying mental model that there's limits to our human capabilities. And if only we can sort of automate the human error out of things, we will be more productive. If you're driving on the I-80, if you saw any of those artisan billboards in the last couple of years, like stop hiring humans with hiring. intentionally misspelt to imply that we are error-prone fleshbags. I think you see a lot of that. Yeah, I just threw in fleshbags to wake up our listeners. You see some of that same sort of Taylorist mindset that's very efficiency-oriented. So you get the sort of trade-off between efficiency and effectiveness that comes down to the values in which the automation approach is grounded. So I think the shearing forces are coming for maybe this sort of Taylorist orientation. And this is challenging because this is like... the paradigm, arguably, like the underlying mental models that may be examined or unexamined that are influencing the values on which the organization moves. Speaker 2 (12:48) You use this term in there, Jidoka, which is a concept that I'm familiar with, but I think a lot of our listeners might not be, that is closely related to how this philosophy of automation was developed at Toyota. And I wonder if you can expand on that. Speaker 1 (13:04) The loom example is the one I go to, like the... idea of you still have humans in the loop, but you don't need to have them constantly. There's an element of there are things that are safe for machines, but there is a point at which the machine capabilities stop and you need to bring the human back in. It's not necessarily any sort of value judgment, just like this is what the machine can do. And there's going to be a point where the machine ends its capabilities and the human needs to take over. Speaker 2 (13:28) Yeah, I guess what I take from that is the idea that you build into the system. The awareness of its own limitations and the awareness of when the system needs