Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?

How I AI

The host conducts a live 'blind taste test' comparing newly released AI models from Anthropic and OpenAI, specifically focusing on Opus 5.5,

Key takeaways

• OpenAI's GPT-6 Sol is highlighted as a favorite for its speed, low cost, and clear, easy-to-read output.
• Anthropic's Opus 5.5 features enhanced guardrails against cybersecurity and biowork but can exhibit a 'conservative scold' personality.

Main topics

• Anthropic Opus 5.5 Review
• OpenAI GPT-6 Sol and Luna Release

Notable quotes

Astra won my heart. Opus 5.5 won my week.

Conclusion

The episode serves as a practical guide for developers to choose model

Transcript preview

Speaker 1 (0:00) I was massively prepared for Anthropic to drop Opus 5.5. I was even prepared for OpenAI to drop another model. I got up early this morning because I had a little bit of early access to Opus 5.5 to record you all an amazing Opus 5.5 review. And guess what? They both, they both landed this morning. So now I am, um, despite having in the can, you will see it later, a great Opus five, five review. I am just going to go ahead and do this one live. And I have never done anything live. So I am going to talk to you all about these new models. There's actually three that came out today. Opus five, five from Anthropic GPT six soul and GPT six Luna. Um, there's like, price wars going on. And I haven't done an episode on the How I AI bench. And I just updated it for these new models. And we're actually going to vibe check this baby live together as a group. And we're gonna go through very quickly what these models are, what they're saying about the models. And then I'm going to vibe check live. We're going to do the blind taste test bench. We're going to surprise myself live. You all can see my complete internal inconsistencies and we're just gonna see how it goes. Speaker 1 (1:30) So quad opus five, five GPT six soul and GPT six Luna, it was able to have some early testing of all these models. So I have a good sense of what I think they're good at, but I did think it was an important moment right now. We've had Astra come out fables been refreshed. Um, I just wanted to completely rethink the how I AI bench. And if you've watched any of our old how I AI bench model episodes, um, they're really focused on two things. PRDs and prototypes. And I just think that these models are getting smarter, more agentic. And so I actually built out a much broader benchmark for this blind taste test. I'm going to walk you through all how that works. And I want to get your feedback on if the comparisons are useful. And we're actually going to blind taste test it live. You all can judge how vibey my vibe check is. This is just how we make decisions about things. So it is imperfect. Does have an LLM as judge in the middle, but But we're going to do it live and we're going to see truly which model I love. But before we do that, let's just go through the highlights. Okay, so these are not the like frontier frontier models. These are not the Astros and the Fables, but Opus got a refresh, Sol got a refresh, and then Luna got a refresh. All of them cheaper and faster. So these are going to be your like daily drivers for common coding and knowledge work tasks. Speaker 1 (2:54) Here you can see the stack rank of how expensive these models are. Opus 5.5 is twice as expensive as GPT-6 Sol. And the pricing wasn't out when I tested these. And this is really going to impact how I think about using these models. Again, Opus 5.5 was cheaper than Opus 5 and Fable 1, to which they compare the intelligence. But Sol, you know, my favorite, my babe. Speaker 1 (3:22) That is I'm excited about because GPT-6 Sol is my favorite and cheap. Okay, so the real thing that they're focusing on, not just cutting the cost, but also cutting on cached inputs and just speed and token efficiency. And so you're going to see both sort of like output. drop, token use drop and cost drop. It's really, really nice. And then caching really great. Especially if you're resending contacts, we've seen a lot of caching savings when we use all these models at chat PRD. And so definitely if you're building on these models, make sure that your caches are optimized and that you're using, taking advantage of that because it can save you a lot of money. And if you don't, as I learned, it can cost you a lot of money. The other thing that I wanted to call out, and again, I hate this cloud-generated presentation, but this is something that I really want you all to focus on, is this left side on Opus 5.5. Opus 5.5 is the first Opus-level model that has shipped with the Fable-level cyber and bio guardrails. And so you should be able to fix your own bugs in Opus 5.5. But you won't be able to do cybersecurity work or biowork, like make good choices. It'll kick you down to Opus 4.8 or block you. So just know that there's those controls there. Anthropic is saying that this is its most aligned model. It's the one that was built when... after like pacing the frontier. So it had external evaluators. One thing that I've seen in Opus that you should expect to see from a lot of anthropic models is it continues to be a little bit of like a conservative scold. You'll see that in my Opus 5.5 review. So, you know, it will tell you no. It'll tell you like, I shouldn't do that. You won't get prompt injected. So that's good. But there are some things here, I think, when you think about the kind of like general philosophy and approaches of these models, how they're set up, how they manage risk, how these companies manage risk. I do think that leaks into even non-risky activity, personality and behavior. And so it's something that I think you should really keep an eye on. I truly have not opened Claude in months. Speaker 1 (5:53) for day-to-day tasks. I've just been like really, really into, um, chat GPT, really into Codex, really into Astra, really into soul. I just find that harness has been more delightful to work with, but really the reason why I got rid of Claude in my day-to-day was between fable and opus five, I could not stand talking. I was constantly prompting Claude, like, can you speak like a human? I was constantly prompting Fable, like, I do not understand what you're saying. The Claude slop, as I say, was Claude slop in. And so just from a personality, ergonomics, joy of use perspective, I really just went full Codex. Now, spoiler alert, having tested all these models recently, I still really love the GPT. models. They make me happier. I find them more enjoyable to use. Everything's a lot better for me, but... I have brought Claude back. So Opus 5.5, just spoiler alert, on the day-to-day of using it has worked its way back into my daily experience. I'm a lot happier with just simply talking to the model. The feedback I gave to the Anthropic team is not annoying. I found myself annoyed zero times with Opus 5.5. And when I thought I was annoyed with... Opus five, five, I was actually accidentally on Opus five. So Opus five is not fun to talk to Opus five, five though, gives me clean bullet points is normal is not annoying. So I would love to see you test it. The other thing is Opus five, five is pretty fast, but not as fast as. soul is. So I was able to test boast and I constantly felt like the GPT models, the Codex harness were way faster. Now I think the way faster came in two categories. It came in actual latency. So I do think that, um, GPT soul was just faster. Um, the second thing that came in is I think part of the way they made Opus five, five, not annoying is they had it shut up. I also just found that it wouldn't narrate its work. And when it doesn't narrate its work, then you get into this loop where you're like, is it like, are you actually working? Are you actually doing it? And so there was like pros and cons to this change in Opus 55 communication where it actually felt slower than I think it was. Also, you know, Claude on high effort. It's like in a... do this. It's going to like spin and do a lot of effort. And so, um, I think that combined with a quieter model just makes it feel a little bit slower. Whereas I felt like soul did the appropriate level of narration. And also it was like enjoyable to talk to, um, let's bop over and do the, how I, I. Viberview. And so you can see here on the left side, I actually really expanded out the How I AI Viberview. So now it does a bunch of things in different categories, and we're going to add to that over time. So it still does messy notes to PRD. It also does personal productivity, inbox triage, and routine replies in my voice. So I'm really testing this model to see if it will speak in my voice. Then, of course, we do front-end coding a lot there because I think it's visual and easy to see. I've added a couple back-end coding, agentic multiset versus back-end feature. I did, again, agentic voice, long-running research, computer use, and then I added some creative features. And so I put them all together. I tested a bunch of models. This not only includes the new models from... uh, open AI and anthropic, but also includes, I think