Testei DeepSeek, MiniMax, Haiku e Gemini na mesma tarefa

Author: Rafael Quintanilha – QuantBrasil · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Testei DeepSeek, MiniMax, Haiku e Gemini na mesma tarefa · Channel: Rafael Quintanilha – QuantBrasil · 38:13 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Software engineering with AI, AI models

Summary

The video compares newer, lower-cost AI models with more expensive options, arguing that DeepSeek V3 Flash and similar models can handle many everyday tasks at a fraction of the cost. The host discusses benchmark results and the trade-offs between intelligence, speed, price, and open availability, then tests DeepSeek, MiniMax, Claude Haiku, and Gemini on the same 404-page redesign. All four produce functional results, but they differ in design choices, completion time, and cost. The host prefers Haiku’s minimalist design, while the independent Opus judge favors Gemini’s page and identifies DeepSeek as the best value.

Key points

The video presents DeepSeek V3 Flash 1017 as a significant update in reasoning, programming, agentic tasks, and visual benchmarks, with reduced response size and execution time.
The host argues that smaller, cheaper models have improved enough to handle a large share of routine work, while the most demanding tasks may still warrant top-tier models.
Artificial Analysis’s Pareto frontier is used to compare models by intelligence index and cost per task; the host notes that speed and other considerations are not captured by that chart.
The host describes a bimodal model landscape: high-end options for maximum capability and cheaper, faster models for everyday tasks, with some mid-tier models offering less compelling value.
A Test Sprite CLI sponsor segment explains how end-to-end tests can help agents validate their work and catch errors.
The practical experiment assigns four models the same 404-page redesign task in isolated worktrees, then uses Opus as an independent judge.
Haiku finishes fastest in the test; DeepSeek is also substantially faster than Gemini, while MiniMax takes the longest in the observed run.
All four implementations are functional, but their visual approaches differ: Haiku is more minimalist, while Gemini adds more content and styling.
The host personally prefers Haiku’s design, while Opus judges Gemini’s page best and DeepSeek the best value, citing its lower cost relative to Gemini.

Chapters

Start	Chapter	Summary
00:00	Why cheaper models matter	The host introduces the falling cost of AI and asks whether DeepSeek Flash can serve as a daily-driver model.
00:30	DeepSeek’s update and benchmark results	The host reviews DeepSeek’s versioning, claimed capability improvements, and comparisons with earlier leading models.
04:39	MiniMax and the growing supply of low-cost models	MiniMax Flash joins the comparison as another inexpensive model, and the host frames both releases as evidence of a stronger value frontier.
07:13	Understanding the Pareto frontier	The host explains how Artificial Analysis compares intelligence and cost, and discusses where DeepSeek, Haiku, and MiniMax fit.
11:26	A bimodal landscape for model choice	The host argues that users can choose between top-tier models and cheaper everyday options, while some intermediate models are less compelling.
17:11	Subscriptions, API costs, and self-validation	The host discusses practical ways to access models, then explains how testing tools can help agents verify their own work.
21:54	Setting up the 404-page experiment	The host describes the task and configures four models to redesign the same page in separate worktrees, with Opus as judge.
27:14	Running the models and tracking performance	The host resolves a model-selection issue, launches the agents, and monitors their progress, speed, and provider usage.
32:29	Comparing the four page designs	The host inspects each implementation and gives personal feedback, preferring Haiku’s simpler design.
35:05	Opus’s verdict and conclusions	Opus ranks Gemini’s page highest and DeepSeek as the best value, while the host reflects on the usefulness of smaller models for daily work.
37:41	Closing remarks	The host asks viewers to share their model choice and promotes the channel, Test Sprite, and an AI engineering training program.

Quotes

does the DeepSeek V 3.1 Flash have what it takes to be your daily driver? — 00:00

we managed to advance the Pareto frontier, the cost-benefit frontier. — 07:13

you need to give your agent the ability to self-validate. — 20:20

The winner for me was Haiku, which was also the fastest — 34:34

Deepseek V3 Flash 1017 being the best value — 35:05

Concepts and entities

DeepSeek V3.1 Flash, MiniMax M3.1 Flash, Claude Haiku 3.5, Gemini 1.5 Pro, Opus 3.5, Artificial Analysis, OpenCode, OpenRouter, TestSprite, Fronteira de Pareto, self-validation

Full transcript (original language)

00:00 And the cost per intelligence for AI models keeps on dropping. This time, the credit goes to the new DeepSeek V 3.1 Flash updates and the launch of the Gemini 1.5 Flash. Two fast, cheap, and small models that deliver intelligence that just a few months ago would have been considered state-of-the-art. I want to run a practical test to confirm if we can actually replace the more expensive models with these cheaper ones and answer the question: does the

00:30 DeepSeek V 3.1 Flash have what it takes to be your daily driver? Folks, let’s start this video right off with the launch of the DeepSeek V 3.1 Flash version 1017, meaning it was just released on October 17th. And this version number is a bit misleading, because the DeepSeek team has a rather strange versioning system. They have the V 3.1 there, and then they do a new release. Instead of calling it V 3.2, they name it—I don’t know—they just put the date here, right? So, at first

01:01 glance, you might imagine that because this new update isn’t even worth calling 3.2, it wouldn’t be that good. But you’d be mistaken to think that way . That’s because the DeepSeek release delivers a substantial improvement in reasoning, programming, agentic tasks, and visual benchmarks, right? So, beyond the benchmark gains, the model reduces execution time and also the response size, right? The response size

01:32 directly influences the price, since most of the cost comes from output tokens, which are actually more expensive, right? So this allows users to complete tasks faster at a lower cost, while also advancing the cost-per-performance on the Pareto frontier. More on that shortly. Well, looking at the benchmark here, we have the new snapshot from 10/17 and the previous one. And you see, for example, look at the jump it has in the terminal

02:02 bench 4.0, right? It goes from 31 to 42 . We’re talking about an increase here of more than a third, right? 35% increase just in this benchmark, surpassing, for example, the Claude 3.5 , without even being the Flash model, right? The large Claude 3.5 and leaving behind the GPT-4o, although without surpassing the Claude 3 Opus. I want to make a specific comparison with these two classes of models now. All right? On Deep SWE, it is outperforming everyone, right? Including Claude 3 Opus. Just a reminder that these

02:32 versions here are the previous versions , okay? We are already on Sol 6.1 and we are already on Opus 5. So, if we look here even at the way the Deepsic team positions the model, it seems it didn’t come to compete, without a doubt , at the top, in the first category. It came to compete with the previous generation of models that were the top. This is very interesting, especially if we consider that the cost has dropped significantly. I wanted to reflect with

03:04 you guys. Look at this, okay? The GPT 5.6 Sol was released on July 9, 2026. I’m recording this video on the 21st, so it’s been about three and a half months since GPT 5.6 Sol was released. Today, the Deepsic V 4.1 Flash basically outperforms GPT 5.6 Sol. If you use it for programming, the Terminal B and Deep SWE benchmarks, it has already been left behind. If we go here even to Artificial Analysis, and

03:35 you put the GPT 5.6 in the comparison, you see that Deepsic would be behind the 5.6 Sol in Reasoning Max. But, as I’ve said a few times here on the channel, I don’t really like using Reasoning Max as a benchmark, because I think most of the time you won’t use it , so you’re not really extracting that intelligence. In 95%of cases, you will use a lower reasoning level. I always used GPT 5.6 Sol on X High. It was my favorite model until we had the updates , before Astra and until the 5.5 family

04:07 of Claude arrived, okay? So I was very happy with the GPT 5.6 Sol X High. And look, if we put the 5.6 Sol X High in Artificial Analysis, we have the Deepsic V 4.1 ahead of it. That’s right , it’s better than the model that 3 and a half months ago was the state of the art and that perhaps a month and a half , two months ago, most programmers used in their daily lives. But we have it being much cheaper and also open source

04:39 . If we look here at the cost per intelligence, we have the Deepsic at a cost of 26 cents, while the 5.6 Sol was at 1.18, right, on X High. In other words, we have it here about five times cheaper, right? The Deepsic is better or at least comparable, but it costs 20 %of the price, okay? So this is very interesting. And besides DeepSeek, we also had the Minimax M 3.1 Flash being released, and it basically enters the

05:09 conversation right here with DeepSeek. So much so that the comparison the folks at Minimax made is mainly with them. So, in the BCH terminal, Minimax would be losing to DeepSeek V 4.1, and in Deep SWE, it is also losing. So, alright, Minimax enters the fight here. I’m mentioning the Minimax M 3.1 Flash because, at the moment I’m recording this video, you can use it for free over at OpenCode Go, okay? So, it’s not costing anything. You can

05:40 subscribe here for $ 10 and basically use it unlimitedly. Of course, it won’t have the best TPS. Of course, we can’t guarantee for how long. It may be that by the time you’re watching this video, this might already be over, but the fact is that it’s a model so cheap to serve that you can get an infinite subsidy here, as is the case with the folks at OpenCode. So it will deliver performance similar to GPT , lower in most benchmarks, but it’s in the fight, right? If we come here to

06:11 the benchmark again, look at the Minimax M 3.1 Flash right here. It’s tied with Haiku and also a little bit behind DeepSeek V4 Flash, right? But these are models, folks, that came here to compete in this class of flash models, meaning models that are typically smaller, have at most 500 billion parameters; you can serve them quite fast, but they are also compared to much larger models, like GLM 5.3 normal, GPT 5.6 Sol, and Qwen. Max and

06:42 even relatively close to what would be Grok 4.7, which is a model considered to have maybe trillions of parameters; we don’t know, but it is certainly a model with over a trillion parameters. So, in fact, there was a greater supply of good and cheap models with these two releases, right? Minimax, for a limited time, even for free, delivers intelligence that just a few months ago would have been considered the best we have to offer as humanity. So that is very interesting. Which also leads us to talk about the Pareto frontier. And

07:13 here is the main reflection I wanted to make with you in today’s video, okay? I’ll do practical tests right away. So stay with me until the end of this video. And if you’re watching and aren’t subscribed yet, go ahead and subscribe to the channel, it really helps us out. Leave a like, and also leave a comment saying what you’ve thought of these two models. But let’s get into it, folks. Why? Because in the DeepSeek announcement they say: “Look, we managed to advance the Pareto frontier, the cost-benefit frontier.” And for us to visualize this, let me go

07:43 over to Artificial Analysis, because they have this chart that I really like . And you will notice it has the Pareto line here, right? What exactly would this Pareto line be? This line tells us which model delivers the best performance for its price range. A quick parenthesis here: what do they consider performance? Performance here is the intelligence index, which is an aggregate created by Artificial Analysis, okay? So, in this aggregate made by Artificial Analysis, there are many things that aren’t just programming, which ends up being what

08:14 most people on this channel are concerned with, right? So, sure, there’s the coding benchmark, but there’s also the humanities exam, so there’s also something aimed at humanities, for other types of tasks that aren’t necessarily code, okay? So, keeping that in mind, know that what they are calling performance, score, or intelligence is this aggregate here. But on the Pareto line, we will have the models that deliver the best performance, according to Artificial Analysis, at that price range. So, for

08:47 example, if we look at the GPT 6.1 Sol Max here, it has a score of 52 and a cost per task of 72. Which means that in its zone here, there’s no one who delivers as well for such a low price. For example, we have the Gemini 4 Argon here that delivers 52—well, it actually delivers 53, but at a cost three times higher and losing in all dimensions to the Opus 5.5 R. For example, look at this. The Opus costs

09:19 1.82 and has a score of 54. The Argon costs 1.99, so a little bit more expensive, and has a score of 53. So you pay a little bit more to get lower intelligence. It’s not worth it. The one on the Pareto line here is the Opus . So here you can keep track of the models that make sense, considering intelligence and price. Note, some things aren’t being said here, for instance, speed. So the GPT 6.1 Sol Max is very slow, right? The GPT Sol is a slow model, and in Max it’s even slower

09:51 , okay? If we look here at this cluster of smaller models, right? We have here, look: the Haiku 5.5, the MiniMax M 3.1, the DeepSeek V 4.1 Flash, and the GLM 5.3 Flash. We clearly have the DeepSeek as superior in terms of intelligence, although it is more expensive here than the Haiku and MiniMax, okay? But the DeepSeek delivers some advantages, okay ? The first advantage it delivers is the fact that it’s open source. So compared to the Haiku, it already wins. That would leave the MiniMax M 3.1 Flash and the GLM 5.3 Flash as well.

10:23 MiniMax, at the moment I’m recording this video, doesn’t have its weights open, so no one can replicate it. So much so that if you come here to OpenRouter, the MiniMax only has its free version being delivered by the MiniMax folks themselves, okay? So here we can’t really judge much yet. If we compare the DeepSeek V 4.1 Flash in this new update with the GLM 5.3, then yes, DeepSeek dominates the GLM 5.3. What do we mean by dominate here? It is smarter and cheaper. It wins in both dimensions. So, rationally, there

10:54 wouldn’t be much reason for you to select the GLM 5.3 here instead of DeepSeek, okay? Oh, Rafael, but the Sol Medium dominates the DeepSeek, look. It is cheaper and has higher intelligence. Yes, in fact, it is on the Pareto frontier, right? The 6.1 model from ChatGPT. It is a very interesting model . I mentioned this, and actually, I’ll put it here for you to watch my GPT 6.1 review later. It’s such a cheap model that it made me rethink if the $ 200 plan was worth it. But what is the main point I want to make here, folks? The

11:26 GPT 6.1 Sol is, indeed, a good model, but you will possibly use it within a subscription. And then, if we compare with subscriptions, we have the Claude models today that are much superior, right? Both the Sonnet and the Opus 5.5 , which in my view is the best model ever created, and I’ve already done the Opus 5.5 review, which I’ll leave here for you to watch too. So, where am I getting at with this? Look, clearly, if we come here to the intelligence index,

11:56 we have three tiers, okay? We have a tier here for the best models, models that are truly state-of-the-art, the ones you can really notice. Oh, the benchmark says so, but is it visibly superior? Are you in doubt about that? Check any comment on my videos or even tweets I’ve made talking about the Opus 3.5, the amount of people agreeing it’s the best model there is. So, we have the benchmarks, we have everyone’s impressions. I think it’s hard for people not to agree that the Opus 3.5.

12:29 It’s the top two, right? Best model we have nowadays, okay? So we clearly have these Claude models, the Sonnet 3.5 is also an excellent model, especially if you consider some benefits. We have it on a separate shelf, okay? I will disregard the Mistral Large here, which is something we can’t test and only appeared here on Artificial Analysis. If we took it out, we would have the three main models being from Anthropic, okay? And that’s with Haiku being the previous version, 3.0, and 3.5 is still coming. We then have an intermediate

13:01 tier of models that, in my view, have become purposeless. I’m talking about GPT-4o, I’m talking about Gemini 1.5 Pro and I’m talking about Mistral Large too. Why? Because if you use them, you will find that they are slow, expensive models that won’t be superior in the aggregate, for example, to Opus. Of course, there will be tasks where they are superior, like Gemini 1.5 Pro, it’s good for text, for example, right? GPT- 4o proved to be very good for 3D

13:31 modeling. I confess I haven’t tested Mistral Large, but I had tested the previous one before, which was also a good model, it didn’t beat the best ones, but it was a good model there. But what is the big issue? This platoon of models here, they are inferior and they don’t have a very interesting cost-benefit ratio. So, you will be paying a high price, relatively speaking; you will be paying relatively dearly for a model that isn’t the top tier, while you could pay more or less the same to have the best model, at least today, which is Opus 3.5, or have

14:03 an even faster version of it, which is Sonnet 3.5. But you can get Opus and Sonnet in the Anthropic subscription today with an excellent limit, right? Including now with the API key that gives you a rebate if you have a Max plan subscription, for example, which I’ve already talked about—I’ll leave it here for you to watch later— and they are still fast. If you look at the TPS, the tokens per second of these models, they are delivering well. So this intermediate group has been somewhat dominated in intelligence,

14:34 speed, and price by the models from Anthropic, in my view. But even though they are dominated, they are still superior, right, to this third platoon, which is the one here that starts from Grok 4.7 and the old generation of ChatGPT, right, the 5.6. I’m even excluding GPT-6, which was OpenAI’s main model for a week, right? So there’s Grok here, then there’s Mimo, which we can’t really use much, right? Then there’s Qwen, and DeepSeek, which just entered now. We have GLM 5.3, Kimi K3, Haiku, MiniMax M 3.1, GLM 5.3 Flash

15:06 , and Gemini 3.8 Flash. The other models have already fallen further behind here, right? So folks, look, we have a first platoon of very good models, we have a third platoon of models that are interesting, fast, and cheap, and we have this middle platoon here that has become a bit purposeless. Got it? So, today I see a bimodal distribution, let’s put it that way. What do I mean by that? When you work with AI, you typically have two types of tasks. The task where you actually want to use the maximum intelligence,

15:38 you want to use the best you have at your disposal, and you have the day-to-day, routine tasks, okay? And this third platoon has become so good, and the DeepSeek V 4.1 Flash 1017 has raised that bar even higher. It has become so good that you can do, let’s say, 80, 90%of the tasks you do in your daily life with it. The proof of this is that it is in many aspects superior to the GPT 5.6 Sol. If we look here at GPT 5.6 Sol on the XY axis, it is superior on Artificial Analysis, as we saw in other benchmarks. And just so

16:09 you have an idea, I even pulled up the launch of GPT 5.6 here, and look at this: the GPT 5.6 Sol Max used to lead the Artificial Analysis coding agent index, right, the programming index. If we look here at the coding index, look at where GPT 5.6 Sol Max is now. Man, it’s in 1, 2, 3, 4, 5, 6, 7, 8, 9, it’s in tenth place here, right? And that’s not even counting several others here, right? Okay? This index here is a bit different because it also varies with harness, so you’ll see that it doesn’t have DeepSeek here, but it fell way

16:40 behind, right? It’s behind several other combinations of models and harnesses. So, this model, which just a short while ago was one of the best we had, can be completely replaced by a DeepSeek, with the advantage that with DeepSeek you don’t even need a subscription, you can pay for it, right , via API normally. Or you could even have a subscription like OpenRouter’s, and you’ll have, man, a lot, a lot of limits to use it by paying $, okay? So, for $ 10, you can use, right, the model

17:11 that basically two months ago was state-of-the-art, okay? And I know that maybe you watching me there are a bit tired; every moment there’s a new model , you have to keep up, what a pain. What I want to say to you here, perhaps even to the detriment of my channel, is that, man, you don’t even need to worry , because you can make this decision. I can pay to have the state-of-the-art. If you have the Claude subscription for $ and you don’t burn through it, you can possibly, today, with $ 100, $ 120, right, use AI in an unlimited, professional way, with the best that

17:42 artificial intelligence offers us today , without needing anything else, okay? That’s it. No need to decide anything else, no need to think about anything else. Put it on Opus, put it on Sonnet, if it’s running out, if you want to do some slightly more advanced orchestration, put it on Riku if you want to implement it with API credits that you’ll get back as a rebate. There’s practically nothing left in the decision-making process for choosing a model that you need to worry about. Now , if you can’t invest that money, there’s also not much to think about; there’s no reason for you to go for intermediate models that you won’t be able to utilize. Basically, you’ll have

18:13 infinite usage using a model like DeepSeek V3/R1 Flash, which is an excellent model, right? So, you go there spending $ 10, $ 20, which means we’re talking about between 10 to 5 times less than if you were to, right, take a leap there to use an Opus 3.5. On a Claude Pro subscription. You can’t do that, man. With $ 20, with $ 10, you either come here to Open Code GO and use it a lot, or add credit to a provider and basically use it in an unlimited way, without worrying. And if

18:46 we come here to OpenRouter, okay? You will be able to see here as a provider, look, let’s select a provider like Cloudflare. Look at that, we’re talking about 10 cents for input, 60 cents for output, and delivering here, look, 70 tokens per second on Cloudflare. If we set it here to Fireworks, which is one I really like, look, 105, comparing it here with Minimax today delivering at 74. I forgot to mention the price here, right? Since Minimax is currently free, the price isn’t showing here, but at their launch, they mentioned it, right? At their launch, the price was here,

19:16 look: 15 cents for input, 60 for output , and a cache of 0.015, alright? So it’s basically the same as DeepSeek V 4.1, right? On Cloudflare, for example, it’s even a little more expensive, right? And for comparison purposes, I also added GPT 5.6, which is what we’ve been using to compare. And look how bizarre the difference is, man! It was 4 per million for input. 20 per million for output, simply absurd. And we have here the Haiku 5.5, the model from Anthropic, also delivering a very interesting speed. This is the fastest

19:47 of all, but here, of course, you’ll either be on a subscription or using API credits over at Anthropic. Alright? Now, it’s no use for your model to be fast if it gives you a result that isn’t correct. That is why, folks, I always hammer home the point that you need to give your agent the ability to self-validate. And that is also why our partner in today’s video is TestPrie. TestPrie is a CLI that allows you to generate end-to-end tests and incorporate them into your agent’s development workflow. Why is that cool?

20:20 Because if you have a model that is less intelligent, and you’re sacrificing that intelligence for a lower price or higher speed, it’s more susceptible to making errors. So, if you give it a self-validation tool, where you’re able to generate that test and at the end of the task verify if that test is passing, then you’ve equalized the capability of a cheaper model compared to the expensive one. So , having the Test Sprite CLI installed, you can, for example, list all the

20:51 tests in a given project, right? And then you can also, for example, get the status of a specific test. So let’s assume we want to check this one, the “ user can change name” test. So we go there, and we execute it. Test Sprite, test run. Pass the ID of that test and look, we now have an execution on the Test Sprite servers in the queue for us to compare. You give this to an agent, it keeps an eye on whether that test will pass. If the test passed, great, it returns to you after finishing. If

21:24 the test didn’t pass, it keeps working, identifying what the problems were, so you have much greater security that when it finishes, what you asked for is not only implemented, but bug-free. I recommend you take a look at the Test Sprite CLI, and if you want to get to know it and test it for free, I’ll leave a link both in the description and in the pinned comment. Wonderful, everyone. For the rest of this video, I want to do a practical experiment with you. I want to validate my thesis that this intermediate group doesn’t make as

21:54 much sense anymore, and that you either go to SOTA or you stay with a smaller, faster, and cheaper model. I’m going to conduct the following experiment. So, those who watched my Gemini 1.5 Pro review, if you haven’t watched it, I’ll link it here; check it out, especially if you have a Google AI plan subscription, okay? There, I did a test where I asked Gemini to change the 404 page, right? That was the 404 page for “Programa AI,” my AI engineering course . And I said: “Man, let’s improve this page here.” I talked to Gemini 1.5 Pro,

22:26 gave it some instructions, made some adjustments, and in the end, for that entire simple programming session, it used 10%of my weekly limit using the Google AI Pro plan, which costs about R $ 100. So, we saw there that the result was okay, but it was slow, and in the end, I wouldn’t be able to use it in practice, I couldn’t use it in my day-to-day work, because if a small prompt, a small task like that, consumes 10%of my weekly limit, I would have at most one or two days of work and then wouldn’t be able to use it anymore. So my point here is the

22:58 following, all right? Since most tasks are typically better-defined tasks, right? I’m not going to go around reinventing the wheel every day. For most tasks, I know where I want to get to and how I’m going to solve them. Maybe they aren’t very long tasks. And imagining that this is what most of us do for most of the day, let’s do an experiment by remaking this 404 page. A simple task, but one you might typically encounter in your work, okay? But I’m going to do this with four models. I want to do this with the new

23:31 version of DeepSeek V3 4.1 Flash, I’ll add the MiniMax 01-1.5 Flash, Qwen 2.5, and I’ll throw Gemini 1.5 Pro into the mix. And to do this together, I’m going to do it in OpenRouter, okay guys? Because here I can run all these models at the same time, right? As I already told you about the OpenRouter subscription, at the time I’m recording this video, you have access here to DeepSeek V3 4.1 Flash, you have the MiniMax 01-1.5 Flash for free. We can use Qwen 2.5 here as well, or you can

24:01 add it via API. And we also have Gemini 1.5 Pro available, okay? So it’s very interesting that we can use Gemini 1.5 Pro inside OpenRouter. Of course, you won’t be able to use it much, okay? If you come here to the limits, look, Gemini 1.5 Pro at the time I’m making this video is at two times usage, which means you can use the equivalent of $ 60 in API costs in your monthly OpenRouter subscription, okay? To compare, DeepSeek V3 4.1 Flash gives you $ 80, only of course, right? Look at the price difference, right? 60 cents off-peak, right? At peak usage

24:34 over in China, $ 1.20 during peak hours . So we’re talking about a model that is 15 times cheaper and has an even higher limit. So you won’t be able to use much of Gemini 1.5 Pro on the GO plan, but for the purpose of our comparison here, it’s enough, okay? So I’m going to run these four tasks in parallel, using a feature in OpenCode 2.0 called Background Tasks. Actually, you were already able to dispatch background tasks in OpenCode, but now in the new update, they can handle much better orchestration. So, there are

25:05 these background sub-agents here. I’ll run these four sub-agents in the background and set the cherry on top here, which is using Opus 3.5. To choose who the winner was. Between these four models, DeepSeek, MiniMax, Haiku, and Gemini 1.5 Pro, they will all try to solve the same task. They will work separately in a worktree. I will pass the result of each one to Opus 3.5 as an independent judge, so to speak. I hope it doesn’t select Haiku.

25:38 And then we’ll check the final result in light of performance, but also price and speed, because that matters in your day-to-day. Wonderful, everyone. I’ve talked too much already. So the first thing I’ll do is set my model here to Opus 3.5, because I want it to be my judge. And then I’ll select it here from the Anthropic provider, which I’ve already connected via API. And then I’ll paste my prompt here. And wonderful. Let’s read the prompt while Opus gets to work. Use background agents to compare four models on the same programming task in the codebase.

26:10 Then DeepSeek V3 4.1 Flash 1017 via OpenCode. MiniMax M 3.1 Flash also via OpenCode. Claude Haiku 3.5 here I’ll use via Anthropic to be faster, right? And also to use my API credits. Gemini 1.5 Pro Contributor also via OpenCode. Create an isolated worktree for each model, all starting from the same commit, and give them the same task: redesign the 404 page, right? Blah blah blah, the same thing we did in the video, right? Of Gemini 1.5 Pro. A simple task, okay? I have no pretense

26:42 of saying that this is complex at all. What I want to do here is verify if, for a typical daily task, we can perceive any significant difference. So we’re going to have these four agents working in parallel, right? And then each of them will have a separate port that Opus 3.5 can use to verify as a judge, okay? It should even take screenshots, compare them, and return a table at the end. So this will certainly take a while. Let’s just check here to see when it dispatches the backgrounds and if it’s really going to do it properly. While Open

27:14 Code works here, you can check all of this in the logs, see? Look at the logs here for Open Code. I can see all the requests being executed. And this cost here is zero. Why? Because I’m using the Anthropic provider. So, basically, I came here to my Open Code console, Connect Provider, selected it, and entered my API key. I got my API key, went to platform.cloud.com, went to the API section, generated one, put it into Open Code, and there it is, I can see

27:44 the request log, executed 1 minute, 2 minutes, 3 minutes ago. So the cost is coming from here, not drawing from my Open Code, but we’ll see it draw from Open Code once it starts using the other models. So this is a cool feature of Open Code to keep track of which requests are being made. Hey guys, it stopped here because it has a question, saying it didn’t find the 1017, but I think it’s because in Open Code it just called it V 4.1 Flash, but I think that is the 1017, right? Because it didn’t put the full name here, like Orama

28:16 Cloud, Fireworks, and DeepSeek did, okay? So I’ll try to keep using Open Code. It’s saying here that Minimax is free and that, anyway, the codebase is private, no problem, it can continue. So I selected it here. Let’s see if this will be enough for it to start the work here. Well, it’s contradicting me here. It said: “Man, I checked the Open Code documentation itself and that’s not it.” So, I don’t know if they haven’t updated it yet while I’m recording this video, or if it’s a documentation error. I think the documentation isn’t updated, but just to be safe, okay? I’m going to say this

28:47 : Let’s use the 4.1 flash 1017 from Fireworks AI. Fireworks is a provider I really like, I have some credit there, so let’s use that. It shouldn’t change anything, guys! This won’t change the model’s capabilities. The most it will change is the TPS, the speed. Look at that, guys, how beautiful. Look at the Open Code V2 background agents running here. Look, DeepSeek V3 4.1 Flash 1017, MiniMax M 3.1 Flash Free, Claude 3.5 Sonnet, Gemini 1.5 Pro Contributor.

29:20 Wonderful. We already have all the agents running here. Now we just wait for them to finish and see what Opus 3’s verdict will be. In the meantime, we can even investigate each of them. See, if we look here at DeepSeek, we can see exactly its thread, right? What it’s doing. Gemini 1.5 Pro, look, it’s failing to make the request here. That’s a problem over at OpenRouter; Haiku is working, MiniMax is working too. Let’s see if it keeps failing with Pro; we’ll have to switch providers, okay? Look at this, guys. I actually couldn’t run Pro via the contributor,

29:52 okay? But I have the Google API connected here, so I already asked it to run the Google provider. And in the meantime, Haiku has already finished, okay? So Haiku was by far the fastest here. And it truly has the highest TPS, the highest tokens per second. So what we saw here on OpenRouter is paying off . Look at the speed difference, right? At least double, right? Compared to its competitors. We aren’t using GPT-4o here. Let me set up Pro here. Alright. Pro, poor thing, hasn’t even managed to run yet. But this TPS here, 22 tokens

30:24 per second, it’s eight times faster than Gemini 1.5 Pro. So we were able to see that in practice. Look at how the speed difference impacts daily life too . Returning here, then, to our experiment. DeepSeek finished in second place. Let’s even click here to see what DeepSeek did. Nice, right? Look at this, guys. 116.9. Tokens per second again. Really quite cool, okay? It finished here in 8 minutes and 56 seconds, okay? Let me go back here so we can see Haiku. Look, it finished in 2 minutes and 52 seconds, 147.3 tokens

30:56 per second. Man, really quite cool. It finished here three times faster than the second-place runner, right? Very, very good. Okay, while the others are running, since they take a little while , I asked Opus to start verifying the ones that have finished, right? So I even said, “Man, go ahead, run it in a background agent right there, okay?” And start validating so we don’t waste any time here. Pretty cool, right? How you can run these agents here. Gemini finished before Minimax, actually, look , it took 12 minutes and 27 seconds,

31:27 okay? These 12 minutes and 27 seconds here, folks, do not count the issue we had at the beginning, okay? So this is the isolated run it had with Google. So it already took 50%longer than the runner-up, right? 32.5 tokens per second here, okay? Really quite a big difference in speed. And look, while we were talking, Minimax finished too. Minimax is the slowest one here. 16 minutes and 25 seconds, although the tokens are relatively fast, okay? There is a detail here: we are running it for

31:58 free within OpenCode GO, okay? So, this slowness here is probably not from token generation, okay? But it’s because the requests are taking longer since they are doing this for free. Obviously, their servers must be quite busy, okay? So I wouldn’t worry so much about this slowness here. I am looking more closely at the tokens per second here. This is for you to also understand that if you don’t pay financially, you will pay with your time. Let’s look at the design, as I’m curious. Look, on port 3101 is DeepSeek

32:29 , on port 3102 is Minimax. On 3103 is Haiku, on 3105 is Gemini. So let’s start here, look. 3101, which is DeepSeek. I’m going to enter a page that doesn’t exist. Look at how it did it, man. Very different from what Gemini had done, actually. In that video of ours. Although there are some things that don’t make much sense, right? Requested address, get, okay? It’s because it’s a 404 page, okay? Let me invent a course here, any course. Done, it tried to do a get here, found a 404, something misaligned here, but there are buttons so the person can

33:00 learn about the AI engineer track or courses, and buttons here too. Found a broken link? Notify on WhatsApp. Doesn’t make much sense. Let’s see here . Eh, on mobile. OK. Cool, right? So, I liked the layout. It put some things here that don’t make much sense. This one here is DeepSeek. So, okay. Now let’s go to 02, which is Minimax. Well, much simpler here. This page does not exist. 404 error. The same primary and secondary buttons. It also put it here in this corner, right? HTTP 404, address not found. Where to go now?

33:31 Here are some more buttons, okay? Funny , it put some very similar things, right? The 3103 is Haiku. And look at that, it did something different, right ? It put here, look, a different footer again, the button with the arrow, the colors, and so on. And let’s go to Gemini. Here’s Gemini, it’s using the same style we have. Some random course address. It also added a WhatsApp link, and it also added other buttons here. Gemini did something with more content, right? It did something denser, I would

34:02 say. So, which one did you think was better here? Funny how they have a very similar variation, right? I’ll tell you that, honestly, I liked Haiku better, you know? A 404 page doesn’t need to have a million pieces of information, right? Uh, this one from Minimx I think was the one I liked least, right? It put a bunch of buttons here in another strip. This one from Deepseek, this side card here I didn’t like. Here from Gemini, man, it looks like a normal

34:34 page, it has too much info. You take a look here, look at the amount of text it has. Two columns, arrow, 40 here, 40 here, button here, link here. Very cluttered, right? I honestly think I liked it. The winner for me was Haiku, which was also the fastest, okay? Look at that, right? It was fast and for me the best. Wonderful, guys. After a long time, Opus finished the report here. So let’s see what it thought. Look, in the words of Opus 5. Gemini 1.5 Pro made the best page with Deepseek V3 Flash

35:05 1017 being the best value, okay? Deepseek’s page is almost as good as Pro, but for 1/5 of the cost. Alright? So they all received the same tasks and they all basically got them right. So none of them made any functional errors . Alright? The main difference is in design and code quality. Plus, of course, if we talk about price and time . Price, guys, by far the most expensive was Pro, which cost. 10

35:36 versus 21 for Deepseek, zero for Minimx , but it would have been about 30 cents if we had compared it to the price they announced. So even more expensive than Deepseek. And Haiku is simply crushing it here at 3 cents and also much faster , right? So, basically, Haiku made the fewest changes, which is why it finished faster. It went for a minimalist design, but Opus judged it as weaker because it created a custom header and footer. So it was more

36:08 inventive here, not following the standard as closely, though for my personal taste, the design turned out better. So it considered the quality of Gemini, PSIC, Minimax, and Claude High here. So what could we conclude here? That in terms of functionality, any of them would have worked. If we already had a design, Haiku would probably be the best. Since we gave it creative freedom, Gemini, which is the larger, smarter model, did more, reused more, and added more style. To my personal

36:39 taste, as I said, I didn’t like it. I thought the page was too crowded, too much stuff, you know? I liked Haiku’s minimalism more, although it did make some mistakes, like using this blue for the main text, which isn’t in the design system. So we have DeepSeek here , which I think was actually my second choice. I would have kept DeepSeek as the second option. And I don’t know if I liked this one from Gemini, not because of the layout, right? It used everything, okay? It added other links, I just thought it got a bit crowded. I

37:11 thought DeepSeek and Gemini were relatively similar, you know? We are talking here about a fivefold price difference, and DeepSeek is even faster , you know? So here is a test for you, nothing scientific, but just so we can get an idea. Do we really need and benefit from these larger models in our day-to-day work? Maybe if we only had DeepSeek, that would already be enough. We don’t even have a spending limit if you’re using it via API at 1/5 of the cost and a very interesting TPS. But

37:41 now I want to know what you thought of our experiment, what you thought of the discussion, and which of these models you would have chosen. Let me know here in the comments. While you’re down in the comments, hit the like button, as it really helps us out here on the channel, guys. Share this with that friend of yours who likes to code with AI and is in doubt about how to choose the best model. I’m sure this video will be useful. If you’re on your phone , don’t forget to like the video here too. I truly thank you for being with me until now. I’d like to thank the folks at Test Pride for partnering with me on this video. Don’t forget, if

38:13 you want to learn more about AI engineering, check out my program, my training specifically on this subject, where we have the AI engineer track, featuring courses on RAG Engineering, agentic programming, and the fundamentals of AI engineering. I’m sure you’d like it too. Take a look at the program; the link is in both the description and the pinned comment. Thanks again for being with me until now, and see you next time. Yeah.