Haiku 5.5: a alternativa BARATA ao Claude Opus MERECE atenção?

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Haiku 5.5: a alternativa BARATA ao Claude Opus MERECE atenção? · Channel: AI ProgBr · 25:06 · Watch on YouTube

Topics: AI content generation, Model inference and optimization, AI models, Local and open-source models, RAG, search and information retrieval

Summary

The video reviews Anthropic’s Haiku 3.5, focusing on its low API cost and whether it can build small games. The presenter compares its benchmark performance with other models, then tests it by playing four generated games: Doom, Mario Kart, Resident Evil, and Crash. Haiku performs impressively in most of the demos, while Step is judged better on Crash. The presenter concludes that Haiku is worth testing for real applications and that cheaper models can work well when used with the right techniques.

Key points

The video presents Haiku 3.5 as a low-cost model and discusses its benchmark results against Sonnet and GPT-4o mini.
The presenter highlights the model’s potential for work tasks beyond programming, such as handling text and spreadsheets.
A video-search RAG application is used to illustrate how an LLM can answer questions using retrieved excerpts.
The presenter says the four game-generation tests used less than 10% of a five-hour quota.
The Doom demo includes working movement, weapons, enemies, doors, and multiple levels, though the presenter notes that some areas are very dark.
The Mario Kart demo has inverted controls at first, but the presenter praises the handling and wins the race.
The Resident Evil demo includes a mansion, puzzles, keys, and combat; the presenter calls it the best Resident Evil game tested on the channel so far.
Crash is the weakest comparison for Haiku: the presenter says Step did better, especially with jumping and crate interactions.
The presenter recommends trying Haiku in real applications and suggests that cheaper models may be sufficient for many projects when used effectively.

Chapters

Start	Chapter	Summary
00:01	Haiku 3.5 and its benchmarks	The presenter introduces Haiku 3.5, reviews its benchmark results, and discusses its pricing and potential as a cheaper alternative.
03:12	A RAG application example	The presenter demonstrates a video-search application that uses retrieval and an LLM to answer questions from video transcripts.
06:23	Testing Haiku with Doom	The first game demo is a Doom-style shooter, which the presenter explores and generally praises despite some dark areas.
09:52	Mario Kart demo	The presenter tests the generated kart racer, gets used to its inverted controls, and wins the race.
12:36	Resident Evil demo	The presenter explores a PS1-style survival horror game, solves a safe combination puzzle, and praises the result.
19:08	Crash demo and comparison	The presenter plays the Crash-inspired game and concludes that Step performs better on this test, while Haiku still does well.
22:24	Final verdict and closing remarks	The presenter recommends experimenting with Haiku and discusses using cheaper models, open-source options, and suitable techniques.

Quotes

And this is where this model really stands out. — 02:09

Excellent result. Very good, man. — 09:52

Best Resident Evil we’ve brought here so far, up to now. — 19:08

But Haiku did really well. — 22:24

If you use the right techniques, you can achieve your results even with much cheaper models. — 25:06

Concepts and entities

Haiku 3.5, Anthropic, Sonnet, Opus, GPT-4o Mini, Step 5 Preview, RAG, quantization, ARC B70

Full transcript (original language)

00:01 Oh, man, he’s a wizard. This week, Anthropic released the new version of their cheapest model, okay? Haiku 3.5. And in the AI landscape, where we see models getting bigger and consequently more expensive, having cheap models that can get the job done is very important. And that’s what we want to see here today. We want to see if Haiku can make little games. Let’s head to the screen. Claude Haiku 3.5. Let’s take a look at their release page, see a bit of the benchmarks, see what they

00:34 claim this model is capable of, and then we’ll see it in practice with our little games to know if it’s a good alternative to the Chinese models. We saw Step 2, for example, which performed very well in our tests. So, let’s see if Haiku gets close to that. Normally here on the channel, when I talk about a model, we always end up looking at it from a programming perspective, right, for programmers. But many people use models not for that , right, but for work, to work with spreadsheets, texts, managing ads, for example. So, various things that aren’t necessarily code. And here we see a big

01:04 gain for Haiku 3.5. So, in work knowledge there, right, it basically more than doubled compared to Haiku 3.0 . From 3.0 to 3.5, we went from 735 in this benchmark to 1620. This puts Haiku 3.5. Ahead of GPT-4o mini. By the way, if you want to see a video pitting them against each other, leave a comment below and maybe I’ll bring it here. One cool thing Anthropic did was to use Sonnet as a base here, right? Sonnet is a respectable model, it’s not Opus,

01:36 right? Right? Opus is their best model and perhaps the best model in the world , but Sonnet is very respectable, except Haiku is much cheaper, and that’s the point here. So 1620 here against 1840 for Sonnet is a chance for many people to actually use Haiku in place of Sonnet, at least for some tasks. Now, what we want to know is this part here, right, which is code. In code, Sonnet stays ahead, stays well ahead here, especially in the terminal benchmark, which will also look at tool calls and so on. But compared to GPT-4o

02:09 mini, Haiku 3.5. Here in Agentic Coding , we have 46.4 against 42.2 for GPT-4o mini and 52 in the high end, right, for Sonnet 3.5. Look, this is where things get interesting, okay? We’re talking about the pricing for a model like this . And price is important, especially if you’re using it via API. And this is where this model really stands out. So, if we look here, let’s start with input and output. So, for input tokens, we can see there are two numbers here. Let me explain what that is. If you’re making a prompt that’s under 100,000

02:42 tokens, in that case, the input will cost 10 cents. If it’s over 100,000 tokens, then the input will cost 50 cents. Now, the output follows the same rule. Below 100,000 tokens. Output at 50 cents; above 100,000 tokens, 50. This is incredible. Do you know which scenario this is incredible for? Imagine you have an application, and in your application, you have a model to provide user support for something. It could be, for example, a prompt enhancer or something like that. By the way, even better. Let me show you an

03:12 example. Let me show you this application I built here. This is an internal application I use mostly when I’m planning a class, but sometimes when I’m making a video or want to get something from a specific video. It’s basically a RAG for videos. So I choose a folder, I transcribe it, I index it, and then I can search for anything. By the way, I’m preparing some of the projects we’ll have in the course, and we’ll probably have something very similar to this. So if you’re not yet a student in our course for devs, take a look at the link below that will appear here or at the QR code that should be

03:45 appearing now so you can sign up and learn a little more about how to develop with AI. So here, I can go to “ search for answers with AI” and I can ask any question about my collection, right, about all the videos I’ve made here. So, for example: how much memory does the Arc B70 have? If I click here on “ask,” it gave me the answer right here. The ARC, it gave it with a K… script gets it with a K, but the Arc B70 has 32 GB of memory. Perfect. Exactly that. What’s happening here? We’re using a RAG, so it’s basically a

04:17 vector search that’s running, and we bring in some clips from the videos that were found and pass them to an LLM model. Based on these excerpts, the LLM model brings us the information we want . In this case here, I’m using a local model, Apple’s own model, but we can see the response is quite simplified and sometimes not the best result. Another thing is that here I ask a question, the model answers me, but let’s say later I wanted to talk a bit more about that answer, wanted to delve deeper. In that case, a more

04:48 intelligent model would make a big difference. Here the local model actually does okay because it gave us the answer. In fact, if I click here, for example, here I have two cards that have 32 GB, I can access the video exactly, okay? Since I have the video title, I can access the exact video and the exact moment I talk about how much memory the ARC B70 has. Here is another video where I probably talk about it too, look. So here was the excerpt it highlighted, look. And if I go down here to the Arc B70, for example, which

05:19 has 32 GB, Arc B, >> it was exactly from here that it pulled it. This is just a small example of where we can have an LLM running inside an application and in this case it will consume much less than 100,000 tokens, okay? Both for input and output, because it’s something simple. Of course, if this conversation stretches on a lot, you might go over 100,000 tokens, but I don’t think so, because 100,000 tokens is a very generous volume. A short while ago, models had a capacity just a little more than that. Only open models still have 260,000 tokens as standard there. So 100,000 tokens is a lot. And in this case here

05:52 you pay 10 cents for input, 50 cents for output, which is very interesting. To give us an idea, the step we saw here, the step 5 preview, which is a relatively cheap model as well, it costs a dollar per 1 million input tokens, 2.70 per 1 million output tokens. So, even if we are over 100,000 tokens, Haiku ends up being cheaper than that. Do we now have a cheap and good American model? How do we check that? With a little game and I separated four here. They are the same four we saw in the Step C preview. So

06:23 here I have Doom, Mario Kart, Crash, and Resident Evil. Resident Evil Step 5 did really well. I liked it. There was a small glitch there where the player was always shooting upwards, but other than that, the game turned out pretty cool. Let’s see Haiko’s result here. Let’s start with classic Doom. But before we see how the game turned out, let’s take a look at how much we spent. And look at that, to make all four games, we didn’t even spend 10%of our 5 -hour quota. So this really is a cheap model. Now let’s see if it worked, right? We have our Doom Tree here, right? Because I asked it to be made

06:56 with 3JS. So, first-person shooter, and does our joystick work or not? Let’s see here, look. Yeah, our joystick is working, yes. Switching weapons here, look. Where is it? Which one shoots? Yeah, no, it’s not shooting. It just needs to shoot. Ah, oh, okay, I found it. Found it. Well, here, look. Awesome . But I’ll go with the mouse. Ah, where did he come from? The door opened by itself. Man, cool little music, huh?

07:28 But the enemies, they come straight at me. The door is opening by itself, man. Didn’t expect that, huh. I thought I would have to open the door there, look . Man, good result so far, man. It’s dark in here, huh? Good result. Ah, look how cool that is, right? Here we have a light point, see? So, we can see everything. Now here we don’t have any light point. And look how dark it is, right? So, okay, that makes sense.

08:01 Let’s go. Let’s go. Look, I don’t need to press anything. Awesome shotgun. I don’t need to press anything to go through the doors. That’s why the enemies are managing to get through too , right? Man, it’s pitch black here, huh? Could have had a flashlight at least, huh? Look, here, here I needed to press it because the door was locked and I had to use the key, which makes sense. Man, these these critters here are look, it’s very dark here, right?

08:40 Very dark. But I cleared the level here , look. Level two. The music changed. Let me put on some headphones so there’s no echo. Ah, damn, it’s a wizard. Ooh, so cool, the wizard, man. Hey, look at the wizard there. Take that, wizard. Where is it? Is there a

09:16 machine gun yet? There isn’t. Hey, that’s not in this game either. No machine gun. I’m screwed on this one. Take that, take that, take that. Ah, I’ll grab that one there. Rocket launcher. Whoa. Take that, take that. Man, even with a rocket launcher this thing is tough, huh? That’s awesome. Oh , let me grab another rocket here.

09:52 Where is it? Everything’s gone here. Yeah, I think that’s everything, huh? That’s awesome. Let’s go over there. Victory. Final arena cleared. We beat it. Level one again. Really awesome. I thought it was good. Excellent result. Very good, man. What did you think? Leave it down here in the comments what you thought about Doom. Let’s go to the next one here. The next one will be this one. Crazy Mario Kart. Uh, damn, huh? The little clouds here. The little clouds. This here looks like a

10:24 croissant. Cool. That’s awesome. Now I’ll, now I’ll use the joystick. Hold on. Now, now it works. Oh, gamepad activated. Let’s go. Where is it? Uh. Ah, it’s reversed. It’s reversed, huh? It’s reversed, huh? It’s reversed. Reversed. I said reversed 10 times, right, so you’d understand it’s reversed. The other day someone said: “ Man, you play really bad.” I don’t know if they were talking about Mario Kart, because it could probably be the other game too, because I play all games

10:57 badly, but eh, playing with inverted controls is kind of hard indeed. Where is it? Which one shoots here? Take that . Ah! Oh, damn! Fell for the banana, geez. Now it’s on. Take that. Man, the controls feel nice, man. The car, look,

11:30 if I don’t steer, it doesn’t follow the line, right? The other day I said, I said that a trash model made a kart that followed a straight line. Man, the guys were arguing with me that it’s correct, unbelievable. Yeah, so this one didn’t commit, damn, two at once. This one didn’t commit that sin. That is a sin. The kart following by itself.

12:03 Sin. Reached the end. I won. That’s awesome. I won. I won. I won. I won. What is this? In first place by accident. Good, good result. I’d say better than the previous one, I wasn’t expecting that. I expected to make this video here saying: “Yeah, this model is good, but the previous one, in this case here, this model is good.” It’s good. Let’s go to the next one to see, for it, Resident Evil. Resident Evil has always surprised me, good things

12:36 always come when we test Resident Evil here. Uh, silent mansion. It’s our PS1-style Resident Evil. This is important, okay? There’s even a difficulty setting here. Obviously I’ll leave it on easy difficulty because. Not hard, hard will break me, man. Easy , not easy. Let’s go with normal. Let’s go with normal. Sound on. OK. Wait, where’s my joystick? It works, oh. It worked, man. This is so PS1, that thing right there, you know? That one, damn.

13:11 Where is it? Where is it? Where is it? Where is it? Where is it? Uh, uh. What happened here? I’m here, look. Ah, I shoot here. Ah, what is this, man? Little pistol. Knife. Where is it? Where is it? How do I change it there? Equip. No, wait a minute. It’s not. Oh, what’s going on here? Ah, ah, okay, okay. No, wait a minute, then. Knife. Equip. I equipped the knife. Man, this interface is really cool, look. I have

13:43 the knife here. Can’t see it, right? Whoa, hold on. You can’t see the the the knife movement, but you can hear it . I’m hearing it. I don’t know if you’re hearing it, but I am. Damn, I’ll I’ll go with the pistol, because the knife won’t work. All right. Look at the pistol. You can see the shot there. Ok. Now I’m going to go down here because I already know there’s a door here and I’m going to be screwed. Get out, get out, get out, get out. Where is it? Whoa, he kicked, man. I think the knife was better, huh. Wait, wait.

14:18 Let, let me put the knife here. Tu my God. Ow, ow, ow. The silence is heavy. Find the master key. Let me see if there’s a door over here that isn’t visible, but we’ll see. There is. Uh, I got stuck somewhere here. I can’t get back. Oh, ended up at some door, see? Geez, it’s so hard to hit this guy. This isn’t right. There’s something wrong here. Equipped the knife. Take

14:50 that stab. Take it. And man, yeah, I’ll go with the little knife, man. Let me go back here just to see if there’s anything. What’s over here? Man, the camera isn’t ideal because you can’t see the whole room, right? Like, for example, I came from here, I came from the other side, I came from here. So, there must be a door here. But but where is it? I can’t I can’t do it. I can’t see. How do I go back? I’m in the library, huh. Oh, I’m just Oh, see? Got

15:25 here, went back. I wonder if there’s a door this way? I’ll go with the little knife, right? ’Cause I’m no fool. I think that’s a door there, right? Let me go back here. Uh oh, I got stuck somewhere here. I’m stuck. Whoa, I was passing through the back there. Okay, I’ll I’ll I’ll go back down here because now, now with the knife I’ve got this. Oh, take that, take that, take that, take that, take that, take that, take that. Ah, now that’s it. With the knife. What what happened here ? There’s nothing here. This is just a wall, right? Oh, there’s one over there . It changed, the camera changed, see?

15:58 The camera changed. So I actually came from down here, and now I’m heading up. It’s locked. I need the lab key. Where is this key, man? This this transition isn’t great, okay? But just a small thing, a small thing. I wonder if there’s one this way? Let’s see if there’s any door this way. Oh, there was a door this way, huh. Take that, take that, take that. I’ll just use the knife, just the knife, just. Oh no, not the dog. That dog is way too tough. Got

16:32 the note. This note is real. Let’s let’s look at the note. Read. Jotted down in haste. The safe combination is the date the butler disappeared. 473. 473 is the date. What kind of date is this, huh? No, there’s no door this way . There is. I didn’t enter any door there, no. I’ll go back. Except the door is locked. I didn’t pick up any key. I picked up a note. Did I pick up

17:03 a key? No, I don’t have a key. I have a note. I’ll go back here. Where is it? There’s not there’s nothing here. There’s a door here. It’s not. I’ll go back there. I’ll try to go to the lab door again, because now I have the note . Ah, now it changed. I can tell by the chest that’s here, right? It changed sides. That means I went to the right place. Oh, it’s locked. I need the lab key. Man, but I have a note, man. Oh, wait a second, wait. There was a chest

17:36 there, wasn’t that a chest in that first room? Is that a chest or is it not? Gonna try to interact with this chest here. It’s not, it’s not a chest. The combination for what is the safe combination. Did you see any safe here? What the heck is a safe? There’s a safe . I’m seeing a safe. Could there be a safe hidden here? No, but something would have popped up there, right? Like , enter the combination, I don’t know.

18:08 It should have appeared. Oh, they actually put the safe in. I can’t believe it. 4 7 3 Oh, the safe opened. I found the key. Now let’s go. Let’s do it. I’ve got the key. Key is right there. The key is right there. I’m using the knife. I’ll go with the knife because the knife has brought me luck. The pistol is hard to aim with. Where is it? I can’t even hit the door. Where

18:38 is the door? Here. There, I hit it. Oh, now it’s happening. Whoa! Oh, man, I got hit. Damn, I got distracted. I got distracted because the Amazon guys got here, I heard a noise outside, I was playing Resident Evil. You hear a noise outside, how do you feel? Scared. So I got distracted, and I got hit. I’ll

19:08 play it again. Not now, not in a video, but later I’ll play this game again. I liked it a lot. Best Resident Evil we’ve brought here so far, up to now. By far, by far. Best Resident Evil. Awesome. Now we’re left with our Crash for the end. Man, Crash is tough. I don’t think Haiko will do well in Crash . There’s no way. I only tried this prompt once with a step and it made almost an endless runner, you know? Like that game, you keep going forward, you don’t even need to move forward or anything. And by the looks of it, it’s

19:39 gonna be the same thing, okay? So, look , where’s my joystick? It’s working. Wow, it’s a lot like the result we had in Later you guys comment that I play badly and I don’t even have anything to say, I’m speechless. I have to agree with you guys. But now it’ll work, now it’ll work. Now I’m gonna give it my all here, look. He jumps twice. No, he spins. Oh, he spins. Where is it? What

20:17 did I have to pass there? Can I jump on top of that? I don’t think so. I don’t remember how to play Crash anymore. Gotta get through. Okay, man. But that, that, that turtle there, that wasn’t for me. I wasn’t supposed to pass through the crate, right? Without stopping, no damage. Whoa. Sorry, sorry , sorry, sorry, sorry. Now it’s happening. Uh oh, something is coming

20:50 behind me. What is going on? Wow, I am, I am particularly bad at this game, huh . What happens if I jump on this one? I don’t, I don’t know what happened. I have to spin into the turtle. I hadn’t figured that out yet. Ah, there we go, there we go. So that was it, look. You have to spin on it, see. There’s the

21:22 finish line up ahead, huh. Take that. Oh, no. Almost got hit, man. Here, so close, so close. I can’t believe it. Now it’s go time. Ah, if I hold forward , I get a boost here. Uh, uh. Now I, now I’ve got the hang of it. Now, now I know. Now, now, now it worked. Now everything is all right. It’s chill.

21:54 It’s under control now. See? See there’s something coming behind me? I don’t know what that is, man. Man, this stuff is getting hard here, huh? I couldn’t even talk there to stay focused, because I don’t want to I

22:24 didn’t want to lose here so close, right? But that’s awesome. I think Step did as well as Haiku. On this one, I’d say it was better, because there were other levels. The jump worked better, the jump on the crate worked better. So I think in this case here, it’s a win for Step. But Haiku did really well. In everything else, Haiku won. I’ll tell you, I wasn’t expecting this, honestly. This is a 50-cent model, 1 million input tokens, 2.50 per 1 million output

22:56 tokens. Obviously, what we’re doing here is just a game, but I want to test this model more in the real world, building real applications. And I suggest that if you already have an Anthropic account, give it a try too, right? Sometimes we get stuck on the best models in the world, well, not us, right? A lot of people do, I try to break away from that, but they get stuck on Opus, on Astra, thinking only those work, only those will do. And sometimes we see a much cheaper model that also works really well. It might not be the best option for every task, obviously, but perhaps for running.

23:28 within an agent or something similar, or to do something you’ll leave running all night and it can keep revising itself. Maybe even something planned by Opus, who knows? I think there are some interesting use possibilities for this model here. Do you think so too, or not ? Do you think this model is no good? Leave it down here in the comments. If you want to see the video about Step 5 Preview, which I thought went very well too, take a look at this link here. In fact, the Step 5 Preview will be open source on the 15th, just around the corner. This means there will likely be other providers offering it at an even

24:00 cheaper price. But if you’re using it from another provider, keep an eye on the quantization, because sometimes you’re paying less, but you’re also getting a worse model that gets lost and takes longer to reach an answer, so it’s not worth it; you lose time and money at the end of the day. Speaking of open source local models, if you want to see a video of my new setup, I’m building a computer here with two ARC B70s, a total of 64 GB of VRAM. Soon it will have a bit more memory than it does now. Right now it has 32, but we’re going to increase that here. I’ve already made a video about my first impressions. Take a look here if you want to watch it, and I’ll record

24:32 another one soon running some really cool models on it. And if you want to break free from the shackles of big models, not be held hostage by Opus, not be held hostage by Astra, or GPT-4, check out my course. there. You’ll learn a lot of things behind the scenes , how the model works, how the agent works, and all that kind of knowledge will be useful when you’re actually creating software, because you’ll be able to better control your harness and better control your model. The consequence of this is that you can use much cheaper models and get the most out of them. I would say that for the

25:06 vast majority of projects, you don’t need a GPT-4, a GPT-Astra, or an Opus 5 . Which are very expensive models. If you use the right techniques, you can achieve your results even with much cheaper models. That’s the case with Rico, for example, a good model, but to use it in the real world, you’ll need to know what you’re doing. And that is the goal of our course. Take a look at it. I’ll end it here, see you in the next video.