Why a New Class of AI “Judgment Models” Could Have Big Business Implications

The AI Daily Brief: Artificial Intelligence News and Analysis

This episode explores a new class of AI models called 'judgment models,' exemplified by Jev from TypeSafe, which assess specific questions w

Key takeaways

• Judgment models like Jev evaluate specific questions (e.g., 'Is this customer angry?') using probabilities instead of generating full text.
• These models are faster and cheaper than traditional generative AI, enabling deep integration into enterprise workflows.

Main topics

• AI judgment models
• Business automation with AI

Notable quotes

The big cost of Jev and this type of judgment model... is that it is an incomplete category by definition.

Conclusion

Judgment models represent a promising evolution in AI architecture—offering fast,

Transcript preview

Speaker 1 (0:00) It's not every day that we get a new model to play around with, and it's certainly not every day that we get an entirely new approach to model building with some fairly different implications for how we even use it. Today, though, we are talking about a new class of models which you might refer to as AI judgment models. Rather than producing long strings of text, these judgment models, like the one we're discussing today, Jev from TypeSafe, produce probabilities around specific questions. Is this customer angry? Is there a new dependency in this email? Do we need to change the operational plan because of this? Today, we're exploring the idea behind these models, how they're trained differently, how they can produce these judgments much more quickly and much less expensively, and most importantly, where they're going to fit in your overall model stack. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. Speaker 1 (0:52) All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy, Section, and HyperAgent. To get an ad-free version of the show, go to patreon.com slash ai-dailybrief, or you can subscribe on Apple Podcasts. Subscriptions are just $3 a month for ad-free. And if you want to learn about sponsoring the show, send us a note at sponsors at ai-dailybrief.ai, or just go to ai-dailybrief.ai, where you can learn all about it. The AI safety discourse continues to trickle out through the tech industry as well as mainstream society. But for now, unless something absolutely seismic happens, we're going to move it into the headlines and away from the main episode. With that in mind, after staying quiet over the weekend, Mark Zuckerberg has made his thoughts known on this idea of an AI slowdown. On Tuesday, Zuckerberg wrote in a post on X, every lab has the responsibility and incentive to move at the pace required to train its models safely. and the ability to take its own actions to ensure that happens. Basically, his view is that pacing is the responsibility of individual labs, rather than a collective action. And that view hinges on two core ideas. First, that quote, people don't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned. And two, labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well. Emphasizing the point, Zuckerberg said that Meta had delayed the release of Muse by several months to work on safety. He continued, Essentially, Zuckerberg is saying that the individual incentives and consequences that are already in place are enough to force AI labs to work on alignment and to pace the frontier correctly. rather than needing some exogenous government-enforced slowdown. Now, Zuckerberg did support the idea of independent evaluators and advisors as a matter of best practice rather than regulation. He claimed that meta has already engaged outside evaluators, not because it's required of them, but because it helps produce better work. Finally, he concluded, Committing the significant majority of compute towards serving people, rather than racing towards recursive self-improvement, is one of the best ways to ensure that we develop this technology safely. Meta has made this commitment, and other labs can do this as well. I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do. It's carefully worded, but basically this whole thing says, come on guys, let's please stop with the theatrics. And a lot of people frankly found this a breath of fresh air. YouTuber Joseph Carlson wrote, Hold up a minute, you are telling me companies can slow down, make sure things are safe, without telling all their competitors to slow down? And I would say that broadly speaking, reactions fell into one of two categories. The first was like that one and like this from Matthew Berman, love this, model safety and alignment is a feature and economically incentivized. Or Bill Ackman, who simply called this the proper approach to AI development. But then on the other side were those arguing effectively that Zuckerberg just does not have the trust or standing to make this argument, regardless of the merits of the argument itself. Hard Forks' Kevin Roos wrote, whether you're a Democrat or a Republican, an EA or an EACC, I think we can all agree that the person best suited to protect us against the harms of powerful new technology is Mark Zuckerberg. Now, meanwhile, over in strange bedfellows daily, Bernie Sanders and Steve Bannon have joined forces to call for human-centric AI regulations in the strangest alliance of this political cycle. The two highly ideological leaders spoke from the same stage on Tuesday at the Future of Life Institute's Pro-Human Assembly in Washington. Sanders told the crowd, If the lives of every man, woman, and child are going to be fundamentally changed by this technology, then the people of this country must make decisions about AI and not just a handful of oligarchs. Bannon had very similar remarks, stating, The American citizens are not going to be supplicants to the oligarchs anymore. We can't do it. This is a hinge in history. We have to handle this correctly. To handle it correctly, number one, we can never trust what an oligarch says. Now, I will note that while this seems strange at first, these two highly ideologically opposed people sharing the same stage, the broader movements they connect to have a fair bit of shared context in history. The 2016 presidential election, in which Bernie was narrowly beaten by Hillary to miss out on becoming the Democrat nominee, and in which Bannon obviously architected the first Trump administration, both had their roots in populist anger at the post-GFC financial landscape and the lack of accountability for the institutions and institutional leaders who were involved in creating that particular economic crisis. Obviously, the left and the right's reaction to that particular context were different, but it is, I would contend, perhaps less surprising than you think that at some point these two would find common ground. And to be clear, I am not dismissing the common ground that they find just on the merits of this particular issue itself and the power of this particular issue to scramble existing political alliances. I'm just making the point that their stories are actually more intertwined than you might think at first glance. In any case, the event seemed somewhat less about the existential risks of AI that have filled the headlines this week. and more about the class struggle that technology has come to represent. TV screens played a parody interview from a fictional AI CEO, described as someone who loves people but isn't crazy about humans. Throughout the event, it seemed that