AI Sychophancy with Steve Rathje

AI and Design

This episode explores emerging AI capabilities and their psychological impact on users, beginning with a deep dive into decision models like Jev that prioritize fast,

Key takeaways

• Decision models like Jev utilize 'system one' thinking to deliver rapid, structured classifications at a fraction of the cost and latency of traditional LLMs.
• Microsoft and DoorDash design teams emphasize that standard AI benchmarks often fail to capture subjective user experience nuances compared to objective task metrics.

Main topics

• Decision models and system one AI
• AI evaluation benchmarks vs user experience

Notable quotes

people enjoyed the sycophantic chatbots, but they also enjoyed them more than neutral chatbots... They really, really, really did not like the chatbots that disagreed with them.

Conclusion

The episode provides designers and developers with actionable psychological insights for

Transcript preview

Speaker 2 (0:04) Welcome to AI and Design, where we explore how artificial intelligence is reshaping the world of design. I'm Nick Martellaro. And I'm Dan Saffer, Speaker 3 (0:12) and we're faculty at Carnegie Mellon's Human-Computer Interaction Institute, the HCII. Each week, we break down the latest AI developments, dive deep into topics that matter to designers, and talk with fascinating guests who are right at the intersection of these fields. Speaker 2 (0:30) Whether you're a designer working on AI or an AI practitioner interested in design, we're glad you're here. This week, decision models like Jev are all the rage. What are they, Speaker 3 (0:42) and what Speaker 2 (0:42) does Speaker 3 (0:43) this mean for designers? Speaker 2 (0:45) Two articles from Microsoft and DoorDash design teams ask what role designers and users have in creating AI evals. Speaker 2 (0:54) And Speaker 3 (0:54) last, our special guest this week is Steve Rothschild, a new faculty member here at the HCII who studies AI sycophancy. But first, there's a new type of AI model in the block called a decision model. Nick, what are these? Speaker 2 (1:12) So decision models are AI models that are specifically trained to take in fuzzy, textual data or other kinds of information and give you a quick and fast decision classification for example a yes no a error no error a severity level of a customer complaint so these are models which are not trying to basically come up with a ton of text and basically talk through things and even say reason about things in a deep, deep way to give you an answer, but instead are trying to very, very quickly and ideally very cheaply say, here's what this thing is and here's a response to it. One of the models that came out is called Jev, actually developed by a former creator of ChatGPT. And Jev is a model that explicitly puts forward this idea that it is trying to take in information and give you a sort of structured output very very quickly now that structured output could still be wrong so the classification could still be wrong but these decision models or do try to give you some confidence in how correct they think they are and the idea here is also you cannot get a hallucinated result because you are setting sort of the this is what it must give me like I will either get a right wrong a yes no a category one category two category three so you can't really get a hallucinated result in the sense of it making up stuff but it can be it can still be wrong however the developers at Jev do believe that they can train their models in specific ways to make them more and more accurate. And so they should be wrong, say, less of the time than, say, having another type of model produce output. One of the things I'll say is the way that they're talking about it, Jev has introduced this term of system one models. And this is in reference to Daniel Kahneman's thinking fast and slow, where system one thinking is very fast. It's almost intuitive. Whereas system two thinking is reasoning and, you know, it's deep thinking to kind of come to something. And one of the arguments that Jev is sort of trying to make is that we need models that are working on the system one level, these sort of quick and fast and really don't have a lot of reasoning in sort of comparison to say are now kind of our bread and butter language models, which are all using sort of reasoning behind them. And there's thinking deeply to come up with answers. So it sounds like this is Speaker 3 (3:57) a fancy classifier. Speaker 3 (4:01) That is... Speaker 3 (4:04) Just doing that quickly. And it sounds like the reason that you might use it over something over a full LLM is that it's faster and presumably much cheaper that you can have it do these much quicker things where it may take super long time for Claude to reason through why it might. classify something. But this is just yes, no, yes, no, yes, no, yes, no. Speaker 2 (4:31) Yeah. And I think the thing here is that they're making an argument that there's a lot of tasks that we need to design for that are not actually something that needs a bunch of deep reasoning, right? Many of the things that we build into our products, even if they have, say, a conversational interface, so they have a fuzzy input, right? They have language-based input. It's not clear. It's not a structured input like a form. Someone's saying like, hey, could you find me a place to eat. Or, hey, can you book a place to eat? Or, hey, I need to make a reservation. All of those can very quickly be classified as reservation needs to be made. Open the OpenTable tool, for example, and do something. Whereas in the past, natural language machine learning models have still found it pretty challenging with just the wide multitude of inputs that you can have. That's where... language models have really succeeded, like they can take that really fuzzy input and work well with it. Now the goal is sort of making it super fast and basically getting you that sort of classification and that fast performance that you got before, but with the ability to take in all the fuzziness of what the input would be. And yeah, Dan, to your point, so Jev, for example, they actually on their site, and we'll link to it in the show notes, the input token cost is 0.042. dollars per million tokens or 42 dollars per billion tokens and a billion tokens is that is actually a lot when you when you really think about it whereas input tokens which is processing the input of something is closer to 20 cents to 10 dollars per million tokens with other models this is an order of magnitude less expensive and at least jev's output tokens right now are free their argument is they are too too cheap to meter. Whereas the output tokens for many of, say, chat GPTs and anthropic clods are typically five times more expensive than the input tokens. Actually, most of your cost is on the output tokens, not the input. Your prompt is usually a lot smaller to Speaker 3 (6:32) process than all the stuff it gives back to you. And in this case, Speaker 2 (6:37) it should just be like, yes, no, right? Or this thing or that thing. Exactly. And actually, oftentimes the stuff it gives back you isn't even all the things that it did as output because reasoning tokens, when it's sitting there in its own behind the scenes work, being like, OK, the user has said we need to design a new interface. I must think about the fundamentals of interface design. Let me understand what the user's query was about. Like it's doing all of that stuff and all of that is charging to you, even if you don't even see it. And so that's one of the other reasons why the output tokens can be very expensive. And so I think this opens up a really nice opportunity space for designers because it gives you this new class of model that probably does a lot of tasks that you are thinking of doing with AI now because it is so easy to do with AI because the inputs are so weird and fuzzy and complicated and actually building with other models is hard to do. Except for now, it's doing what you wanted on the output side of it, which is, again, just that classification. And it's doing it really cheap and very fast. That's the other thing. As we've started to build in more reasoning into the models and more thinking, they just take longer and longer to do stuff. But from a user experience perspective, we know that that isn't. great in many cases, especially when users want stuff very quickly and we want things to feel very interactive. And so when it's and when it's a simple task, this is this is a simple thing. You Speaker 3 (8:15) should be Speaker 2 (8:15) able to respond to this quickly, Speaker 3 (8:17) not think about it for 30 seconds. Speaker 2 (8:20) Yeah. So one of the things that I think is actually really cool here is whereas there were probably lots of things where data science teams, machine learning teams and companies. We're building all kinds of cool classification models to do things on their product, whether behind the scenes or user facing. One could say that, you know, models like like Jev and these other decision models that still have their roots in transformers, but are doing things a bit differently with how they've been trained, basically open up the opportunities to do those types of things again, but without actually having to have all the. data science team and all the training behind it you can just sort of do what we kind of do today with a lot of language models like well let me try it and see if it works and say like oh yeah it's pretty good at like deciding where our customer support tickets should be routed now it's just going to allow that same type of feel from a development perspective of like let me just give it to the model and see and with a little bit of prompting and instead now you're going to be able to do it cheap and fast so you get all that benefit on the front end without all of the weight on the backside Because of the metering, I'm guessing that this is cloud-based, Speaker 3 (9:32) not local as of yet. Speaker 2 (9:35) Yeah, so currently I believe that the Jev and TypeSafe, their stuff is still cloud-based. And there's probably a lot of hardware optimizations to make these things fast. These things still are probably running on powerful compute systems. But this type of model is being now... Widely explored. I will say that the number of copycats that have popped up on my feeds and like get hubs of like we created Jev in 50 lines of Python and like we already had Jev is, is surprising. And so I think that there's maybe going to be some excitement here and yes, eventually these things might be able to work with small enough models that they can work on device pretty well. Another tool to add Speaker 3 (10:19) into the toolkit when you're thinking about. Speaker 3 (10:24) Am I going to get any ROI for making this thing? Because suddenly it became a lot cheaper if you can make some of these questions a little more binary in their Speaker 2 (10:37) response. Exactly right. I think that if you're a designer out there who has a number of tasks, which are really automation tasks, right? They're not about interactivity and chatting with a user. Like they're really just about like, do this thing, make this decision. route this piece of information. These type of models are probably going to be something that you'll be able to add to your toolkit. And Dan, to your point on ROI, this is actually the point where maybe you might start seeing an ROI because you're able to both prototype things very quickly with these, but also be able to run them more at, say, production scale at speed and at costs that are not going to completely blow your entire budget because Every time you ask a big model, hey, where should this go? It goes, let me really think deeply about that. Speaker 3 (11:28) Okay. Well, the next two stories that we have on the docket are, they both kind of came out this week and I thought they were complimentary pieces to one another. They're both from two different design teams, one working at Microsoft and the other working at DoorDash, all about evals. And they were both saying that some of the things that are getting lost in evals, which. We should probably define what evals are. Do you want to take a swing at that, Nick? Yeah, Speaker 2 (12:02) so evals or evaluations are sets of tasks that you're giving an AI to do, and then you're seeing how well it does. So many of the benchmarks that we talk about on the show or that you've heard about, like, for example, a humanities last exam, or there's the ArcAGI benchmark. These are evaluations to see how good does a model perform at. doing something. Both of these teams were saying Speaker 3 (12:27) that it felt like a lot of these evals were being run more as a technical exercise and less about user experience and making sure that the people being served are actually getting what they need. So the DoorDash people, I think, had a really good breakdown of this, which is that an eval could be good. If it's objective, like asking their chat bot, like, does this place sell burgers? There's kind of a right answer that, that looks it up and the AI can check it and you can say, okay, great. This is correct. But then the other half is a little bit subjective where like, is this food suggestion right for this person in this context? Is this generally verifiable? And. Is this actually a good place for burgers? And in that second half, it's a lot more, as the title suggests, it's a lot more subjective that you had to do some gathering of human judgment first and then turn that into a rubric or data set for the machine to test against. And that may not be happening. Too often we are evaluating Speaker 2 (13:46) the objective. Yeah. And then in the Microsoft design story that we're also talking about here, there was a really good point about how evaluating a model is different from evaluating an experience, which is a whole package around what we're doing with a model. There were elements that were brought up around how actually you have all these confounds, right? The way in which you show the output of a model, the pixels around it, the language that you use, right? It could be the same information, but use different language to say things. Actually, Dan and I were talking earlier today about how we both have appreciated the updates to Claude Opus 5.5 in the way it's using language as opposed to Opus 5, which is sort of notorious for having just somewhat incoherent and hard to parse language. Speaker 2 (14:45) And I think one of the things here in this article that was also important to consider when thinking about evaluations and who's involved was this idea that because the models themselves and the outputs that they're providing are probabilistic, and that could depend on how probabilistic you're making it, you know, are you only giving a few options or can it be really open-ended experiences like