Well-Tested, Easy to Use: How to Measure Any Product's Usability (w/ Philip Kortum)

The Design Psychologist | Psychology for UX, Product, Service, Instructional, Interior, and Game Designers

In this episode, host Thomas Watkins interviews usability expert Phil Kortum about the System Usability Scale (SUS). They explore the histor

Key takeaways

• The System Usability Scale (SUS) has remained a gold standard for over four decades due to its reliability and efficiency.
• Usability is defined by three pillars: effectiveness, efficiency, and satisfaction.

Main topics

• System Usability Scale (SUS)
• Usability Metrics

Notable quotes

Being able to summarize something as a number is like a holy grail of standardization.

Conclusion

The episode underscores that standardized measurement tools are essent

Transcript preview

Speaker 2 (0:01) Welcome to The Design Psychologist, the show that helps you use psychology to design better experiences. I'm Thomas Watkins, your guide to becoming Speaker 1 (0:12) a more powerful, psychology-informed designer. Speaker 2 (0:17) Imagine you've just conducted a usability study and you calculate the results and you run into the meeting room and you say, we've got a usability score of 78. And people look at each other and they look at you and say, okay, is that good? Being able to summarize something as a number is like a holy grail of standardization. Humanity has been through this many times before, agreeing on what a foot means, or what a pound means, or what time is, and so on. But how do we get there, to something reliable and quantitative, with something as subjective as human experience? Furthermore, just because something is a number doesn't automatically make it objective. We've got to know, where does this metric come from? What standardizes it? What does it measure? How do we know it's reliable? Does this metric continue to mean the same thing in different situations? Today, I'm excited to share a conversation I had with Phil Cordom, who I'll introduce momentarily. Together, we'll explore the System Usability Scale, or SUS, with questions like, Why has this 10-item questionnaire survived for more than four decades? How does it compare to something like the Net Promoter Score? Why is it that asking several questions produces a better measure than simply asking one? How do we know what SUS scores like 50, 70, or 90 actually mean? When can researchers modify the scale without undermining it? And with the SUS, we're asking people questions. So isn't that subjective? How is it that the SUS ultimately gives us an objective value? By the end of this conversation, you'll have a much clearer understanding of what it means to measure usability well and why good measurement can give designers a common language for not just talking about but understanding the human experience. So let's have a listen as I introduce and speak with today's guest. Speaker 2 (2:29) Welcome to The Design Psychologist. We've got a great guest for the show today, Phil Kortum. He spent his year in research roles, technical roles at famous places like AT &T Labs and before that Raytheon. Today he is professor of psychological sciences at Rice University. He focuses on human factors, usability measurement, and applied psychology, known for bridging the... rigorous side of usability with real-world applications of product and system usability. He has co-authored a bunch of really important research papers in the field, namely an empirical evaluation of system usability scale, SUS. And so we'll be talking about that 2008 paper and a bunch of others. So Phil, welcome to the show. Thank you. Glad to be here. Yeah. So let's start off with getting into, you know, you've worked with tons of human factor systems in the 90s, 2000s, and so on. Before we get into the importance of the system usability scale, give listeners a backdrop of kind of the nature of the work, the nature of the problems, and what gave rise to SUS and the need eventually to kind of validate it empirically. Speaker 1 (3:45) Sure. So I think... For those of us in the usability community, one of the big things that we try and do is we try and figure out, and it's a simple question, is this system or device or product usable? And for really simple systems, that's pretty easy. If I give you a can opener and say, hey, is this easy to use? That's easy for us to assess. As I get more and more complex systems, it becomes more difficult. And so if I'm judging a cockpit, how can I judge the usability of that? If I'm judging an entire battleship or guided missile frigate, how do I determine if that's usable? And so typically, we try and use measures that are defined by ISO 9241 Part 11, which are effectiveness, efficiency, and satisfaction. And the two that we really hang our hats on from a data standpoint are effectiveness and efficiency. And effectiveness just says, can I do the task? If I ask you to use the can opener to open a can, can you do that? And then I can simply measure, were you able to do it? I can also take other effectiveness measures, like how many errors did you make? How many attempts did you have to do before you were able to open the can? I can measure other things such as... How many times do you have to call the help desk? How many times did you reach for the manual? Efficiency is also an important measure for lots of products, and that's, could I do it in a timely way? And so almost always, we just put a stopwatch to that and say, it took you X number of minutes to complete this task. That number by itself is not particularly meaningful, but it's meaningful as a comparison if you're doing iterative work, right? have version one, version two, version three, or if I'm trying to compare your can opener and my can opener, am I faster with one or the other? But the raw number doesn't mean anything. It might take me two minutes to do a task that's something simple, and it might take me an hour to do a complex task. And in both cases, those might be considered very efficient numbers. We just don't know. And then satisfaction is a little bit... more subjective. We're just trying to understand if people find the experience enjoyable. I think what happened is a lot of people wanted to tie some numbers to the satisfaction number. And we started looking for surveys or instruments that we could use that would measure the subjective usability that we would use as a proxy for satisfaction. And so This started to happen in the late 80s, early 90s, when people started trying to put some numbers to that. And the reason that subjective usability numbers were beneficial is it allowed you to make more direct comparisons across products. It allowed you to understand how things were changing over time. And it allowed you to think about being able to measure without specific reference to a task. So if I, let's say I'm trying to measure Microsoft Word. And so the task that I might measure would be. create a new paragraph, and then bullet that paragraph. If then I want to compare that task to adding the table of contents, it gets much more confusing about whether the time metrics and the success metrics match. But if I can come up with a numeric estimation, then I can at least say, well, that task was subjectively easier than this task. Speaker 2 (7:31) Right. Okay. So just to jump in there, because so much great stuff here. Just to summarize, we've got, you know, different ways of assessing usability. We are building things. We want to make sure that people can actually use them. It's actually making people's lives easier and better. So the three effectiveness, efficiency, satisfaction that you've laid out, effectiveness, can I do it? Efficiency, yeah, I can do it, but can I do it in a somewhat timely manner? And then there's the satisfaction. And typically we say, well, okay, well, that's pretty subjective and everyone can have their own opinion, but that has some usefulness in it. The direct comparisons across products, you could look at things over time and it's free. It's a little bit abstracted from a particular task, which classical usability tests. require it to be a test. So you have some ability to say, how good is the usability of this system or Speaker 1 (8:29) product? That's exactly right. Yep. Speaker 2 (8:33) Yeah. Okay. So, so, so going from there in, I'm glad you pointed out that, you know, this kind of stuff started off in the eighties because you look at 1996, it seems kind of late, right? It, right. For, for that to be the one. So, so, um, you know, continue your previous thought, but why, this 1996 uh sus scale and you know what's special about it Speaker 2 (9:02) If this show has sparked new ideas for you or helped you think differently or deepen your approach to design, consider leaving a quick review or rating on Apple, Spotify, or your favorite podcast app. It helps people find the show and it enables us to attract more great guests that I know we can all learn from. And honestly, it's always great for me to hear what's working for you. So if you can leave a quick review or rating, thanks for listening. And now we'll get back to our episode. Speaker 1 (9:39) Sure. So there were lots of efforts to come up with different kinds of instruments that would measure usability. And so Jim Lewis was doing tons of this work at IBM. He developed the after scenario questionnaire and the post-system usability questionnaire. He had done a lot of work in this. There were also a number of proprietary surveys that had been developed, the SUMI and the WAMI. for measuring software and for measuring websites. And of course, at the time, I was at AT &T and we were looking for a survey instrument. And we had a couple of criteria. And one was we wanted it to be freely available. We were testing lots and lots of things. We had 15 or more members of technical staff doing usability tests all the time. And we didn't want to have to pay a fee. every time we wanted to have it scored. And we also wanted to, lots of those proprietary systems also score those in a black box fashion in that you provide the data to them and then they give you a score back. And so you're not really sure what's happening under the hood. And we wanted to be able to do our own psychometric evaluations of that. But really important for us. was we wanted to make sure that we could use the same instrument to test all the different things that we were developing at AT &T at the time. And so we needed a survey that was going to be technology agnostic. It didn't really matter. And so we landed on Brooks' system usability scale, which was published in 96, but he actually started working on it and developed that all the way back in 1984. And so it had been out and around. But when he finally published it, it looked like it met all the criteria. It was short enough that we could administer it quickly. So there was a light burden on both the user and the researcher. It was totally agnostic. And just to give you an idea of how agnostic it is, we've used it to test things like websites and cell phones and software packages and voting machines and medical devices and textbooks. Face masks during COVID. I mean, we have really, really used this instrument to measure the usability of everything. And so we started using it and decided we were going to use that instrument to gauge the usability of every single thing that came through our lab. And so that's what resulted in the 2008 paper where we took 10 years worth of that data and collected it up. and then did some analysis to understand the properties of the system usability scale. And it turns out those properties are quite good in terms of measuring what we think it's measuring, being really reliable, measuring it the same every time we want to measure it. And so we kind of stuck with it. And I've continued to use it over the next 20, 25 years after that initial 10-year period. For those very reasons. There's a lot of, there, there've been some additional entries into the field that, that I really like. For example, Arnie Lund's use survey, usability, satisfaction, and efficiency. And it's, it's, it's a very nice survey, but at the time it was a little bit too long. It was 30 questions instead of 10, like the sus. And it was. It hadn't been validated at all. And so we didn't jump on that. At this point, there's so much archival data on the system usability scale that it really makes sense to keep using it, even though some people like to do modifications for very, very specific reasons. But for the most part, the SUS has proven to be a really nice utility. And so it's one of the... I think, I think Brooke himself wrote that it was the most widely used instrument to measure usability. And he, he had, I think he collected 25,000 studies or something ridiculous. Speaker 2 (14:07) So Speaker 1 (14:07) lots of people are using it. Speaker 2 (14:09) Yeah, that's insane. And one of the things I like about the story is it kind of highlights the. extremely applied nature of the profession where he'd been using something since 1984 and people had heard of it, I'm sure, at plenty of conferences had talked about it. But it's like, okay, I need to write this up. And it's like, you know, 12 years or so later. So the fact that you guys were just testing tons of things within AT &T, can you give us an idea of the types of things that you would be testing or evaluating? Speaker 1 (14:42) Sure. And so this was a great time at AT &T because the internet was a brand new thing. And so there was tons to do. Mobile technology was also a brand new thing. This is when cell phones had just started to come out. And so we wanted to make sure that all the cell phones that we were selling, we weren't developing cell phones, of course, at the time, but we wanted to make sure that the ones that we picked from the marketplace were the most usable for our customers. And so we did lots and lots of comparisons between this handset by Nokia and this handset by someone else. And so that was one thing. We were developing self-installation kits for, again, broadband internet was just becoming, had just been invented and we were starting to field that. And so we were developing kits that would allow people to do that installation in their own home. And so we did that. We were developing internet appliances. There was a ton of software that was being developed to do things over the internet that we hadn't thought of in the past. For example, backup of PCs, which of course seems perfectly mundane at this point, but at the time it was considered wild and new. We were testing because we worked for the telephone company. We were testing services that were delivered over the telephone. And again, some of these seem quaint at this time, but we were building interactive yellow pages that you could access on your phone. The interactive voice response, the dreaded press one to do this, those were in their heyday at that time. Of course, those are still. widely and broadly used. But we were developing all kinds of new technologies behind that to help those be better. And so we were doing evaluations of usability on all of those. And so there was just this whole host of very different kinds of things that AT &T was working in, all of them in some way related to communication. But at the time, the definition of communication was exploding. And AT &T was moving away from being, hey, we just, all we do is deliver dial tone to your house to becoming the internet provider of choice, becoming the mobile communication, the mobile phone provider of choice, and still providing dial tone to the house. And so it was a marvelous time to be alive and to be collecting all this data on this huge variety of Speaker 1 (17:30) And it's one of the things that when I came back to the university, I brought that idea that this