The Question You Should Ask Before 'How Many Users?' (w/ Laura Faulkner) | S2 E3

The Design Psychologist | Psychology for UX, Product, Service, Instructional, Interior, and Game Designers

Thomas Watkins interviews veteran UX researcher Laura Faulkner about the limitations of the widely cited 'five users' rule in usability testing.

Key takeaways

• Laura Faulkner's 2003 article challenged Jakob Nielsen's 'five users' usability testing rule, demonstrating it often misses subtle but critical problems.
• Faulkner argues sample size must be driven by risk assessment and the potential impact of missing a problem, not fixed magical numbers.

Main topics

• Sample size in usability studies
• Risk assessment in research design

Notable quotes

"It's not about finding fewer and fewer. It's about finding the right ones."

Conclusion

The episode provides UX professionals with a pragmatic framework for j

Transcript preview

Speaker 2 (0:01) Welcome to The Design Psychologist, the show that helps you use psychology to design better experiences. I'm Thomas Watkins, your guide to becoming a more powerful, psychology-informed designer. How many users do you actually need for a usability study? Is it 5 or 10 or 20? It's tempting to want a rule that simplifies this decision for us and just gives us the answer. But what if the more important question is, what happens if your research misses something? Maybe five people reveal the obvious problems. But what about the problem that only appears for user number 6 or 10 or 19? And what if that less common problem is the one that causes someone to lose their work or abandon the product? or in high-stakes situations, leads to something much worse. Today, I'm excited to share a conversation I had with Laura Faulkner, who I'll introduce momentarily. Together, we'll explore questions like, what is the question of sample size really about, outside of being a magical number in whatever field you're studying? How can researchers become better decision support partners for their business? How can we balance the sharing of research results early to build trust rather than carefully controlling the data? How do we develop a strong point of view without confusing evidence with a personal opinion? And as AI and synthetic users enter the research process, what can they reveal and where do we still need direct human data? By the end of this conversation, you'll have a more thoughtful way to approach research, not as a set of rules to follow, but as a disciplined way to reduce uncertainty and help people make better decisions. So let's have a listen as I introduce and speak with today's guest. Welcome to The Design Psychologist. We have an excellent guest for the show today, Laura Faulkner. Laura has spent more than two decades at the intersection of psychology, technology, business, and human-centered design. You might know her from her widely cited article in 2003 that challenged one of the most repeated claims in usability lore is that all you need is five users. to conduct tests, and that's it. And with rigor and clarity, she showed why that rule of thumb isn't always enough, and she shaped how many of us think about sample size, research design, and evidence-based practice. She's held senior research leadership roles at places like Rackspace and AWS, where she's led UX for massive infrastructure projects like AI and ML platforms, cloud compute, and dev tools. What sets Laura apart from that is everything else is just not just the technical skills. It's her deep commitment to the research as a driving force is a, you know, for marketing, for changing stakeholder minds. She's created frameworks for stakeholder alignment. She's written brilliantly about designing better research reports, and she continues to mentor the next generation of UX researchers with a mix of realism, insight. and guiding with heart. Laura's work has asked us to think critically about how we plan studies, how we communicate findings, and how we show up as researchers in complex product environments. Laura, welcome to the show today. Thank Speaker 1 (3:47) you. Thank you. So Speaker 2 (3:48) let's start off with like you had your... background in experimental psychology. You're now very much a business person. Can you tell us a little bit about the time where you were making that shift? Speaker 1 (3:59) It's actually been almost three decades that I've been in user experience. I just celebrated my 29th year. 28th year, I don't know, something like that. I started my tech career on March 7th of 1996. And to that date, I'd always been passionate and been studying the human sciences. My degrees are in human sciences and sociology, anthropology, psychology, with minors in biology, communications, and statistics. So really, I just loved human beings. And I loved studying, and I loved being able to put things into frameworks. But the other thing was, is that as I went along in my academic career, I discovered that I didn't want to be theoretical. I didn't want things to just be out there and write things that sat out there and that people talked about. It's like I wanted my work to move the needle in some way. And I didn't know if something like that ever existed. Well, along the way, I was also supporting myself. As a legal assistant working in law offices, and this before automation, or early days of automation, like a typewriter with a disk in it that you could actually save. It was very revolutionary to the IBM Selectric with your memory. You could save things in. And in that time... I just happened to be in an office in a big tech area in Houston, just right across the street from NASA. And so there was an IBM desktop computer back in the file room where I, as a receptionist, was putting the files. And there was a piece of plastic over it. And I knew what it was. It's like I'd seen it advertised in the news and my eyes had lit up like, they can put a computer on your desk? That's like, I grew up with whole rooms full of computers that had, that sent people to the moon, you know? And so I, I went into the senior partner's office. So this actually like dates back more like 45 years. And the, I asked, I went into the senior partner's office. I said, is that a computer? And he said, yes. Do you want it on your desk? And I said. Yes. He said, everybody else is afraid of it. They don't want to touch it. Speaker 1 (6:39) So I began making things easy because that was what I wanted to do. And I can't even tell you, it's like all of us who find our way into user experience, no matter what your specialty is, from visual to interaction design to, you know, full-on fancy Figma designer, whatever that you do down to, you know, and into the deep research or whatever. You know, even UI developers, it's like there was something about being... making things easy for real people. And we all, like, have that commonality no matter which pathway we came in. And so I began to develop, you know, to develop things, to create databases and use technology and create systems and then write manuals for it. You just did it, whatever, to make that easy. Well, by the time I left there, like, two years later, The entire office was automated. All the processors were automated. Everybody had a computer. Even the attorneys could do it. And that I continued to do that is to automate. Then I landed in Austin, came to graduate school in theoretical anthropology, still supporting myself in the legal field, but trying to find my way. And I really wanted... I wanted something else, and I ended up changing careers to being a tech writer for the University of Texas at Austin in the Applied Research Laboratories. It's a defense lab that serves Navy, Air Force, Army, Marines, and also served the Royal Australian Air Force and Royal Canadian Air Force. And over those years, I was right in the heart of R &D technology. And on that first day, so that's why I date March 7th, 1996. That was the first day I went in as a tech writer. And here's how everything changed is that first day I'm sitting there and they're showing me, okay, here's the software and here's the manual you'll be updating. And here's what you're doing. You're going to write instructions. You're going to update these instructions for this. And I'm looking at this interface and I'm thinking. Speaker 1 (9:01) Okay, if I took out these two windows, moved this here, changed that there, and called that a search, I could write this in seven steps right now. I'm going to have to write 12 pages. Speaker 1 (9:13) And I didn't want to write 12 pages. And I thought, okay, this has got to be a thing. And then I found out that day that they would support going to conferences. So I found one, and it was Society for Technical Communication in the spring of 1996. And they, so I signed up for it in Seattle. And there was a workshop called Usability. And that word just rang, and I thought, this has to be it. This has to be the thing. And Jakob Nielsen's Usability Engineering had just come out. It wasn't a paid field. You had to be doing something else. But we were all drawn to it, and we started applying what makes sense in practice. And honestly, that still makes sense today. So the fact that it emerged as a career, that was wonderful. I'm proud to be on the front end of that where I got like a paid job for it and got some benefits from that and then got to, you know, develop that as I went along to other places. And then finally got to be applying for jobs as that was like the big, when I first got to apply for a research. leadership position or for a full-time researcher position. Those were just exciting. And then I led a design and research actually for Pearson Clinical Assessments. And so we were doing, as you were a unicorn then, you did design and research. I will say research is my first love. So when design became its own practice, I chose and I thought I'm going to let the folks who are really drawn to that. be the experts in that. And I'm going to step back and I'm going to go or step off to this other path and stay in research. So that's how it all happened. And, you Speaker 2 (11:00) know, Speaker 1 (11:01) that kind of journey is still valid today. It's like we're drawn from all sorts of places, but we have a thing that we want to do. Speaker 2 (11:08) Yeah, it really is still like that part today, you know, where you come from so many different directions and adapt with her. whatever skill you're using. So the technical writing of like, hey, I need to solve this immediate real problem in the office by applying some, you know, guidance with words on how to do these things. But then that goes over to the designing. And then like human factors professionals back then, it was a combination of research and design skill sets a lot of the time in the 90s, especially. Speaker 1 (11:38) And Speaker 2 (11:39) then so when. Then we fast forward a little bit to your 2003 article. And, you know, that became one of the big splashes, the first big splash that you made, I think. People started kind of knowing who you were, but it was addressing a real issue because some statement was made by Jacob Nielsen based upon a little study he did. And people interpreted it in certain ways. And even to this day. It is the case that I run across people regularly when it's time to do a study. I've heard it, you know, more than a few times of, oh, well, you know, just test a handful of people and they don't have an understanding of sample size and they use those heuristics to just replace having a understanding of sample size. So in terms of that moment, that sort of 2003 moment. Tell us about that. Speaker 1 (12:39) So what happened, what was happening is, you know, we were hearing that and there's this argument about sample size. And really, it's the wrong question. It's like, it's neither about how, you know, you need to do a bunch or you need to do a little bit. It is about what is it, what is it you need, what is it you're going for? What is it you need to solve? So five, the actual way to interpret that. is 5 is perfectly fine for finding the glaring issues that any 5 users are going to find. The problem is, let's say that we have a set of usability problems. And this is actually how I did the research, is that my intern, the late David Wick, he and I created, we intentionally created a piece of web software that we designed usability problems into. But we did a couple of things. One, we were designing a known set of problems because all of that work and all of that thinking that Nielsen and other, that assumption is made on, is all based on we already know how many are in there, so therefore we know when we found 80 % of them. That's just false. You can't even say that because it's an unknown set. And even we couldn't know exactly how many we had built in, but at least we knew that these were usability problems that we had built in. So how many of those are we going to find? And then the other thing is, I said this key word like glaring, obvious. It's like you're going to find those pretty much no matter who you test. Probably because you already know they're a problem and your own heuristic analysis would have found that if you're an expert in any of these fields or even not an expert and just a user yourself. So you're going to find that you're going to know that. So you're going to find what's already repeating. So it's not it's no longer relevant that after five you get this law of diminishing returns. You find fewer and fewer problems. It's not about finding fewer and fewer. It's about finding the right ones. Which ones, which problems are going to have an impact on the user? Sometimes that is, it's going to impact everybody, so we better fix it. So that's where a gross measure such as five or something matters and where it's useful. Or when you're just trying to convince, like you've done