Claude Haiku 5.5 vs Local Qwen 3.8 27B: Is Local AI Still Worth It?

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Claude Haiku 5.5 vs Local Qwen 3.8 27B: Is Local AI Still Worth It? · Channel: Zero to MVP · 07:43 · Watch on YouTube

Topics: AI evaluation and benchmarks, Claude, Software engineering with AI, Local and open-source models, Privacy and data protection

Summary

The video compares Claude Haiku 5.5 with a locally run Qwen 3.8 27B on four tasks: a Python expense-splitting utility, two changes to a markdown editor, and processing confidential financial documents. Haiku completed the tasks much faster and passed every hidden test, while Qwen matched it on the utility and financial-document tasks but made mistakes on the editor work. The local model used far less money for the large coding task, though it took much longer and missed a significant bug; the video also highlights the privacy benefit of keeping personal documents on the computer. The creator concludes that cloud models are preferable for major changes to large projects, while local models remain useful for smaller tasks and sensitive data.

Key points

Both models used the same Pi Agent tools, maximum reasoning level, and tests that were hidden until each model reported completion.
On the expense-splitting utility, both models passed all 32 checks; Haiku took about 3.5 minutes, while Qwen took about half an hour.
For the word and character count fix in Textodus, Haiku passed all 13 hidden tests; Qwen failed one by counting each Korean syllable as a word.
On the file-renaming and link-update task, Haiku passed all 13 hidden tests and 530 existing tests, while Qwen passed 9 of 13 hidden tests and missed a bug affecting links in open documents.
Haiku’s large-project task cost $1.44 because 58 requests exceeded its 100,000-token threshold; the creator estimates it would otherwise have cost about 32 cents.
Qwen’s coding run used about 400 watt-hours of electricity, estimated at 15 cents at the creator’s rate, excluding hardware costs, but took almost two hours.
Both models passed all 32 checks on the financial-document task, including correctly calculating totals and keeping sensitive personal data out of the output.
The creator favors cloud models for serious changes to large projects, and local models for smaller tasks and work with confidential data.

Chapters

Start	Chapter	Summary
00:00	Comparison setup and expense-splitting task	The creator introduces the Haiku 5.5 versus local Qwen 3.8 27B comparison, explains the test setup, and finds that both models pass all checks on a Python utility, though Qwen takes much longer.
02:06	Textodus fixes and the harder link-update task	The models tackle word and character counting and then automatic link updates, where Haiku passes the hidden tests and Qwen misses a bug in open documents.
05:11	Costs and confidential financial documents	After comparing the coding-task costs, the creator tests both models on tax deductions and sensitive documents, with both passing all checks while only Qwen keeps the documents local.
06:43	Conclusion: when to use local or cloud AI	The creator concludes that cloud models are faster and better suited to major code changes, while local models remain valuable for smaller tasks and privacy.

Quotes

So, in terms of quality, the result is the same, but the difference in time is already significant. — 01:35

The project builds, all the existing tests pass, but one of the core features works incorrectly. — 04:41

And in my opinion, that’s one of the strongest arguments for local models. — 06:43

If I need to make serious changes to a large project, for now, I’ll use cloud models. — 07:13

Concepts and entities

Claude Haiku 5.5, Qwen3.8 27B, Pi Agent, Textodus, Anthropic, llama.cpp, Obsidian, modelos locais, Janela de contexto e tarifa por tokens, testes ocultos

Full transcript (original language)

00:00 Last week Anthropic released Claude Haiku 5.5. It’s their fastest and cheapest model, just 10 cents per million input tokens. At prices like that, an interesting question comes up. Why run local models on your own computer at all? I decided to find out. I took Haiku 5.5 and a local Qwen 3.8 with 27 billion parameters and gave them the same tasks, from a small Python utility to fixing bugs in my real app and working with personal financial documents. The results turned out to be

00:32 interesting. On some tasks, Qwen did just as well as Haiku. On others, it took several times longer and missed bugs it didn’t notice itself. Let’s see what happened. For testing, I’m using the Pi Agent. Both models work with the same tools without any extra extensions and at the maximum reasoning level. I wrote the tests in advance. The models can’t see them and I only run them after the model says it’s done. Haiku runs through the Anthropic API. Qwen runs on my desktop with a Radeon Pro with 32 GB

01:03 of video memory. The model is running in Llama CPP with Q4 quantization. I set the context to 192,000 tokens. So, let’s get started. The first task is to write a Python utility that splits shared expenses between friends and works out who owes whom how much. At first glance, nothing difficult, but I hid 12 traps in the spec. For example, money can’t be stored as floating point numbers. Leftover cents have to be distributed by a specific rule. And at

01:35 the end of the document, there are changes after review that cancel some of the earlier requirements. Let’s start with Haiku. It finished in 3 and 1/2 minutes and spent less than 4 cents. All 32 checks passed. Now Qwen. It also passed all 32 checks, not a single mistake, although it worked for half an hour, almost nine times longer. So, in terms of quality, the result is the same, but the difference in time is already significant. Now, let’s move on to a real project. I

02:06 have a desktop markdown editor called Textodus. It has more than 20,000 lines of code, including some in Rust. The first task is to fix the word and character count. Right now, Chinese text is counted as a single word, and some emoji, like a flag or a family, count as several characters. I gave both models the plain text of the issue with no extra explanation. Haiku fixed everything in 8 minutes, cost 16 cents. All 13 hidden tests passed. Qwen also

02:36 mostly solved the task, but one test failed. It decided to count every Korean syllable as a separate word, even though Korean separates words with spaces. As a result, a two-word phrase turned into seven words. A small mistake, but still a mistake. Now, a harder task. Textodus has a problem. If you rename or move a file, links to it from other notes stop working. I want them to update automatically, kind of like in Obsidian. There are quite a few nuances here.

03:06 Links inside code blocks must not be touched. If a document is open and has unsaved changes, you can’t just override it on disk. And when a folder is moved, relative links have to be handled correctly. Okay, Haiku first. It worked for 22 minutes and took 81 steps. All 13 of my hidden tests passed, and all 530 existing tests in the project. But, the cost turned out to be surprisingly high, a dollar 44. The thing is, Haiku 5.5 has

03:38 a threshold of 100,000 tokens per request. If you go over it, the whole request costs five times more. And an agent working on a large project keeps sending the model the whole history of its work. In my case, 58 requests went over that threshold. Without the higher rate, the task would have cost about 32 cents. So, this is worth keeping in mind when you use Haiku encoding agents. Now, Gwen. It worked for almost 2 hours and took 130 steps. And after about an hour

04:10 and a half, the context got close to the limit. The agent had to compact the history from 181,000 tokens down to 27,000. That alone took about 10 minutes. But in the end, the model finished the job. All the existing tests pass. The linter and the type check, too. Gwen even wrote its own tests, and they pass as well. Now, let’s check it with my hidden tests. Nine out of 13. Two of the mistakes are fairly minor, but another one is a real bug. If

04:41 a document is open in the editor and contains a link to the renamed file, Gwen doesn’t update that link at all. The model was right not to overwrite open documents on disk, so that unsaved changes are kept. But then it forgot to update the links inside the open documents themselves. And the model didn’t notice this problem itself, and reported that the task was done. In my opinion, this is far more interesting than an ordinary compile error. The project builds, all the existing tests

05:11 pass, but one of the core features works incorrectly. A pretty dangerous situation. Now, about the cost. During the run, the graphics card used about 400 watt hours. At my rate, that’s about 15 cents of electricity. Of course, not counting the cost of the hardware. For comparison, Haiku cost a dollar 44. But given the difference in time and in the quality of the result, for a task like this, I’d go with the cloud model. And the last task. This one isn’t about

05:42 programming. I prepared a folder with financial documents, bank statements, receipts, the accountant’s emails, and documents with confidential information. The models have to collect the tax deductions for 2025 into a table and prepare a note for the accountant. And there are a few traps here, too. For example, the bank labeled the gym membership as a health expense, even though there’s no tax deduction for it. On the pharmacy receipt, only the prescription drug can be deducted, not

06:13 the whole purchase. And have to make sure that the full social security number and bank card numbers don’t end up in the output documents. Haiku did it in a minute and a half, less than 2 cents. All 32 checks passed. Now, Qwen, 6 and 1/2 minutes, and also 32 out of 32 checks. All the totals are calculated correctly. All the conditions are met. No extra personal data in the output files. So, the result is the same, but

06:43 there’s one important point here. When I used Haiku, I sent all these documents to Anthropic’s servers, and with Qwen, they never left my computer at all. And in my opinion, that’s one of the strongest arguments for local models. So, what did we get? Haiku passed all my tests without a single mistake and worked much faster. All four tasks cost about a dollar 65. Qwen also showed good results, especially in the first two tests and with the financial documents.

07:13 But on the hard task with lots of code changes, it fell noticeably behind Haiku. So, here’s the conclusion I came to for myself. If I need to make serious changes to a large project, for now, I’ll use cloud models. They’re faster, and in today’s experiment, Haiku did that kind of work better. But for working with personal documents and small tasks with clear requirements, the local 27 billion parameter Qwen is already quite enough, especially since API prices for small tasks are now so

07:43 low that saving money on its own no longer looks like a convincing reason to buy an expensive graphics card. But being able to work with your data without sending it to the cloud is a completely different matter. So, I’ll definitely keep using local models, just not for every task. If you’re interested in tests of AI models like this, subscribe to the channel and to my weekly newsletter about IT and AI. The link is in the description. See you soon. Take care.