Qwen 27B on 6GB VRAM...

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Qwen 27B on 6GB VRAM… · Channel: Prompt Engineering · 14:13 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, Local and open-source models

Summary

The video tests Prism ML’s Bonsai 2, a roughly 6 GB compressed version of Qwen 3.8 27B, against the full-precision model. Bonsai performs similarly on short tasks, supporting its claimed benchmark performance in those settings. In longer agentic builds, however, it repeatedly loops through tool calls and fails to produce working pages, while Qwen completes several builds. The conclusion is that Bonsai is useful for chat and short coding tasks on local machines, but the full model is a better choice for long, unsupervised agent work.

Key points

Bonsai 2 uses ternary quantization, representing weights as -1, 0, or +1 with shared scale factors, and applies a rotation to smooth outlier weights.
Prism ML claims Bonsai 2 reaches 98% of the full model’s performance across 20 benchmarks; the presenter notes that this average uses maximum thinking effort.
The comparison ran Bonsai locally through Prism ML’s llama.cpp fork and Qwen through OpenRouter, with thinking turned off for both.
On short tasks, the models had similar results: both answered the long-context questions correctly and both failed to produce a working one-shot Linear clone.
In six longer builds, Qwen scored 35 out of 60 and produced working apps, while Bonsai did not produce a working page.
Bonsai’s traces showed repeated tool calls and extensive file searching; in one case, it ran the same tool 114 times.
The presenter notes that Prism ML’s paper reports lower scores on longer agentic benchmarks: about 76% on Terminal Bench 2.1 and 75% on Sweep Bench Verified.
The video recommends reading agent traces to diagnose behavior and suggests Prism ML’s llama.cpp build for agentic use on a MacBook because it caches conversations between steps.

Chapters

Start	Chapter	Summary
00:00	What Bonsai 2 is and how it is compressed	The presenter introduces Bonsai 2’s performance claim and explains its ternary quantization and weight-rotation approach.
04:44	Benchmark setup and short-task results	The video describes the local-versus-API test setup and reports that Bonsai and Qwen perform similarly on simpler tasks.
08:57	Long agentic builds reveal a gap	Across six multi-step builds, Qwen completes working projects while Bonsai repeatedly fails to produce a working page.
10:32	Inspecting the agent traces	The presenter examines logs showing Bonsai repeating tool calls and compares this behavior with Qwen’s own, different mistakes.
12:06	What the results mean	The video relates the findings to longer benchmark scores and concludes that Bonsai suits short tasks better than long, unsupervised agent work.

Quotes

That’s exactly what Prism ML is claiming with their new Bonsai 2 model. — 00:00

98% is an average over 20 benchmarks — 04:44

it didn’t produce a single working page across all six tasks. — 10:00

Bonsai made 153 tool calls on the ISS tracker task. — 11:02

For long unsupervised agent work, I would still reach for a full model. — 13:41

Concepts and entities

Bonsai 2, Qwen3.8 27B, Prism ML, ternary quantization, Pi, llama.cpp, MLX, Terminal-Bench 2.1, SWE-bench Verified

Full transcript (original language)

00:00 Can you get 98% performance of a 27 billion parameter model in under 8 GB? That’s exactly what Prism ML is claiming with their new Bonsai 2 model. Now, the question is does it actually hold up in real-world testing? So, I put it to a test in my own benchmark, directly comparing it with the Qwen 3.8 27B. And on simpler tasks, it is really close. It basically is a tie. But on longer agentic tasks, where the model has to plan, write code, and check

00:31 its own work, and then keep going from there, it is a very different story. So, in this video, I really want to show you what kind of impact compression can have on a model, and where it actually shows up, and how you can spot it yourself, because we can learn a lot from it. Okay, but before that, we need to talk about what exactly Bonsai 2 is. Because the way it is made is very interesting, and it’s a part of the whole story that I want to cover. So,

01:02 Qwen 3.8 27B at full precision stores every weight as 16-bit number. So, we’re talking about 54 GB if you use 16-bit floating-point precision, is more than memory most laptops or consumer GPUs have. Bonsai 2 cuts that down to about 6 GB, which is roughly nine times smaller. Now, usually when you want to make a model smaller, you quantize it. A good mental model of quantization is

01:33 rounding. You take a model that’s already trained, and you round every weight to a number that needs fewer bits. It’s cheap and fast, and you see this for most models out there. But the problem is if you round too hard, the model starts losing its intelligence, is in the weights. Now, an ordinary 2-bit version of the same Qwen model is going to be about 7.3 GB. And according to Prism ML’s own table, it keeps about

02:03 88% of the benchmark scores. Then, there is a smarter version of that idea, which I covered in my 12-star video. The insight there was that you don’t quantize everything. You keep the part of the model that matters most at higher precision, and you aggressively quantize the rest. So, it’s really selective quantization, but it’s still rounding a model that’s already been trained. Now, Bonsai 2 takes a very different path. It starts from the pre-trained Quen 3.8 27B, and then it’s trained into

02:37 a new format. So, instead of just being rounded into it, Prism ML says that it was trained on Google’s TPU, but they haven’t published the method, the data, or how much compute it took. So, part of it is simply black box. But, we do have the final weights and the format in which it is quantized. Now, Bonsai uses uh something called ternary quantization, which converts every weight into one of three possible values: -1, 0, or +1.

03:09 In this case, the big positive weights become +1, big negative becomes -1, and the small ones are rounded to zero. They also use an additional scale factor to preserve how big weight was. So, every group of 128 weights share one scale, and the real weight is basically that scale times the ternary value. Now, you need about 1.7 bits per weight in order to store those ternary values plus the scale factor, which is about nine

03:40 times smaller compared to 16, which is the actual uh full precision for the Quen 3.8 7B model. They also use an additional trick. In any standard neural network, a handful of massive weights will completely blow out the shared scale for entire group of those 128 weights. To fix that, before turning everything into ternary, they apply a rotation matrix to smooth out those huge spikes across all the values. Once those outliers numbers are spread out, fitting them into those three

04:12 distinct value gets a lot cleaner. Now, they claim that this helps preserving the intelligence of the model. So, a naive two-bit version retains roughly 88%. Bonsai shrinks the footprint further while hitting a 98% rating on key benchmarks. So, the idea is that you get a smaller file alongside better performance, which kind of goes against what usually happens. Now, one important detail that I want to mention that 98% is an average over 20

04:44 benchmarks, and it was measured with the models thinking turned on at maximal effort. Keep that in mind because we’re going to come back to it in my own experiments. Now, most of these benchmarks and most of the comparisons that you’re going to see are single shot. You give the model prompt, it gives you an answer, and you score the answer. But, that’s not how we are using these models in agentic setup. In an agentic loop, the model is going to read files, write code, run it, take screenshots, find bugs, fix those, plan

05:17 around it. And, in order to do all of these things, they are going to take sometimes dozens or hundreds of steps. So, the real question that I wanted to test was does that 98% hold up in an agentic loop? Okay, so here’s the setup that I tried. On one side, we have Bonsai 2 running locally on my Mac through Prism ML’s own fork of llama.cpp. On the other side, we have the original quant 3.8 27B at full precision, which is exactly what Prism ML compares it against. Now, I can’t run

05:48 that on my MacBook, so we are using it to open router. Both of these models are running inside Pi, which is the agentic coding harness that I’m using within the same tools and same prompts with the same instructions. Now, for both of them, I kept the thinking off. And the reason was that I wanted to get a really clean baseline for the model itself. And the thinking off actually shows up in harder tasks, too, which we are going to look at in a minute. One more note,

06:20 Prism ML also have native support for Apple’s MLX framework, but in my testing, it was much slower compared to Llama.cpp, so that’s why I went with Llama.cpp. Okay, so there are two different kinds of tasks that I ran. The first set is short, kind of thing people usually test. The second set is long agentic builds, and for those, I didn’t just eyeball the results. So, every build comes with a checklist of specific requirements, and a a script checks each one automatically. And this is

06:51 important, so I’m not just using the this looks good to me. There is an objective test where uh say we have a certain number of features that we want the agent to build, and then the grader actually checks whether it built those features or not. Okay, so let’s start with the simpler tasks. The first one is a drawing task. So, the prompt was draw an octopus playing a drum kit as an SVG with strict limits on colors and shapes. Gwen scored seven out of 10, Bonsai scored six based on the rubric that I provided. Both of

07:24 them are pretty close. Next, an analog clock that has to show an exact time from uh the URL that I provide. Now, this is harder than than it looks because the checker measures where every hand points down to a few degrees. In this case, both of them actually struggled, so Quen got three, Bonsai got four. Then a debugging task, so I gave both models a breakout game with six planted bugs. Four were reported by players, and two

07:54 you could only find by reading the code. Interestingly enough, Bonsai fixed five out of the six and scored 8.5, Quen scored seven in this case. Then there are two short agentic tasks. So, one is a messy sales spreadsheet with four hidden traps, like duplicate rows and a currency mix-up. The other is a broken refund report in a database. Both models caught every trap and fixed the report, which is actually pretty encouraging, right? So, for on simpler tasks, seems like both of them are head

08:26 to head. And then there’s a long context test. So, I gave both models a real code base of about 100,000 tokens with some facts planted in it and asked seven questions. Now, both of them got every single one right. And the last one was one-shot linear clone in a single file, and both of them produced a page that crashed on load, right? So, same failure on both sides. So, seems like for simpler tasks, both the successes and failures looks very similar.

08:57 So, I would say that 98% claim actually holds up for simpler tasks, even if you have thinking off. But, all of these were tasks that you can finish in one answer on or in just few steps. Okay, so let’s talk about round two, the agentic builds. Uh the second set is six real builds. So, first is a black hole simulation with gravitational lensing, then a pirate ship battle in one bit max style, and then a voxel sea turtle on a coral reef. Uh another one is an ISS

09:27 tracker that places the station on a 3D globe with a real cell positioning, and then a launch page built from a research brief. And the last one is a full linear clone with team cycles, keyboard shortcuts, and undo. Now, in this case, the model is required to actually plan, write several files, run them, look at screenshots, and fix what is broken. So, Quinn scored 35 out of 60. It built the linear that passes 19 of the 24 requirements. Lunch, it

10:00 passes all 11 of 13. On the ISS tracker, uh it’s nine out of 13. On a turtle scene, it’s nine out of 15. Now, it struggled with the hardest graphics like the physics of the black hole and the strict pixel rules of the pirate scene. But, it did ship a real working app in Bonsai completely crashed on these. So, it didn’t produce a single working page across all six tasks. On five of them, it never even got to a page uh in its project folder.

10:32 Okay. One thing that I want to point out. Uh the Bonsai is running on my Mac. Quinn is running through the API. So, I actually went to an earlier set of runs I did on my DGX Spark about 10 days ago where Bonsai had hours for every task. Now, it failed in exactly the same way there as well. And I think this is where it’s really important to look at why these failures happen. Uh so, I was keeping all of the logs and found some really interesting things. On difficult field tasks, it usually

11:02 gets stuck on simply looking around. So, Bonsai made 153 tool calls uh on the ISS tracker task. 150 of them were those were just looking around, uh reading files, listing files, and searching. And one other thing was looping. So, I I noticed that on a lot of these harder tasks, it would run a tool, get the results, and then it would run that tool again and get the same results repeatedly. For example, in one case, it ran one tool 114 times. Now,

11:35 when you compare directly to Qwen, the worst repeat across all of these six different projects was six. So, Qwen had its own problems. On the pirate scene, it spent about 40 minutes chasing one syntax error on its and its own code. And it has a habit of rewriting whole files instead of editing them. But, those are mistakes of a model that’s building something bonsai Qwen is way different. It can write perfectly fluent code, but it doesn’t seem to notice that

12:06 it has already done something. So, on harder task, it basically never moves on. Okay. So, after running the experiments, I was actually trying to figure out what exactly was happening. So, I noticed that Prism ML’s own white paper points to the same thing. Now, they cite this 98% number on 20 shorter benchmarks, but their paper also reports two long agentic benchmarks, and those two are not in the average. On Terminal Bench 2.1, bonsai keeps about 76% of the original score. On Sweep

12:38 Bench Verified, which is built from real GitHub issues, it’s about 75%. Now, to be fair, it’s still an impressive score for the model of its size, but it shows that it’s a good model for shorter task. Okay, so what exactly are the takeaways from this, right? So, first, matching on benchmarks doesn’t mean matching on actual agent work. On short tasks, bonsai basically ties up to the full model, but on long builds, it couldn’t really finish anything, at least in my experiments.

13:09 Second, if you run agents locally, read the traces because they’re are going to tell you what exactly is happening with the agent. And this can also help you improve your agent to loops. Third, if you want to run this model on a MacBook, I would highly recommend to try out Prism ML’s Llama CPP built for anything agentic because it caches the conversation between steps and the MLX version at least doesn’t as of the recording of this video. And finally, where does that leave Bonsai 2? So, I

13:41 would say for chat, quick coding help, and short tool calls on a laptop, it’s actually a pretty great model. And the compression actually works there. For long unsupervised agent work, I would still reach for a full model. So, compression enables us to run these big models on local machines, but in general, it doesn’t really hold the true intelligence of the model. Keep that in mind if you’re reaching out for a quantized local model against a bigger full version. Anyways, I hope you found

14:13 this video useful. Thanks for watching, and as always, see you in the next one.