Bonsai 2 27B Locally: Near-Lossless Compression in a 9x Smaller

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Bonsai 2 27B Locally: Near-Lossless Compression in a 9x Smaller · Channel: Fahd Mirza · 17:08 · Watch on YouTube

Topics: AI agents, Software engineering with AI, Model inference and optimization, Local and open-source models, Multimodal models

Summary

The video introduces Bonsai 2 27B, a ternary-compressed model based on Qwen 3.8 that is presented as fitting in under 6 GB while retaining much of the original model’s benchmark performance. The host explains its compression and attention techniques, then tests it locally on codebase debugging, HTML generation, image interpretation, and multilingual translation. Bonsai misses the planted time-zone bug, fails to make the latte simulation work despite repeated fixes, and produces mixed results on the vision task. The translation test appears to get stuck, leading the host to judge this version less impressive than its predecessor.

Key points

Bonsai 2 27B is based on Qwen 3.8 and is claimed to be roughly one-ninth the size of the original, with 98% of its benchmark performance.
The model compresses most weights to ternary values (-1, 0, or 1), uses a hybrid attention approach, and supports a 262,000-token context window.
The host runs the model using a custom llama.cpp fork and reports a little over 12 GB of VRAM use with multiple request slots; reducing the slots lowers consumption.
In the wildlife-monitoring app test, Bonsai fixed a different chart discrepancy but failed to fix the planted UTC/local-time bug affecting the 24-hour gauge.
The model generated a latte-making HTML interface, but its sequence button did not work, even after multiple attempts to fix it.
The vision test received a generally positive assessment, though the host says the original Qwen model performed better.
The multilingual translation test appeared to stall and hallucinate, with the model getting stuck partway through its reasoning.
The host concludes that Bonsai 2 was less impressive than the previous version, despite the model’s size and performance claims.

Chapters

Start	Chapter	Summary
00:01	Introducing Bonsai 2 27B	The host presents Bonsai 2’s size, claimed performance, context window, and plans to test it locally.
01:03	Installation and how the compression works	The model is downloaded, and the host explains its ternary weights, hybrid attention, and long context window.
03:38	Serving the model and testing codebase debugging	After discussing VRAM use, the host asks Bonsai to find and fix a hidden time-zone bug in a wildlife-monitoring application.
07:16	Model internals and debugging results	The host describes the rotation technique behind the compression and shows that Bonsai fixed another issue but missed the planted bug.
10:23	HTML generation and benchmark claims	Bonsai creates a latte-making interface, which looks promising but does not work, while the host puts the published benchmark claims in context.
13:58	Vision and multilingual tests	The model gives a fairly good image-based response but appears to fail when asked to translate a sentence into multiple languages.
16:03	Final assessment	The host concludes that Bonsai 2 is less impressive than the previous version and invites viewers to share their thoughts.

Quotes

A bonsai tree isn’t a small species. — 00:01

But we are not going to fall for the hype. — 03:08

Bonsai actually found a real discrepancy in the 14-day trend chart and fixed it. But that wasn’t the bug we planted. — 09:19

So you see this one is a fail. — 13:58

So far, it is just hallucinating. — 16:35

Concepts and entities

Bonsai 2 27B, Qwen3.8, llama.cpp, Hermes Agent, quantização ternária, atenção linear, rotação Hadamard, Fahd Mirza

Full transcript (original language)

00:01 A bonsai tree isn’t a small species. It’s a full-size tree pruned and trained over time until it’s the same tree just shrunk down. That’s the idea behind this whole model family. And today we have got the second version, Bonsai 2 27 billion. It’s built on quen 3.8, which is a 27 billion parameter model that normally needs 54 GB just to load. Bonsai 2 prunes that down to under 6 GB,

00:33 roughly a ninth of the size, while claiming to hold on to 98% of the original’s benchmark performance. It runs a 262,000 token context window, uses a hybrid attention setup, mixing fast and precise attention, and ships under Apache license. We already have covered the previous versions which were quite impressive 5 months ago. So it has taken them around 5 to 6 months to come up with this second version and what’s best

01:03 it is based on our beloved Quen. So let’s get it installed and test it out. I am going to use this Ubuntu system. I have massive GPU card but don’t worry about it. I’m going to show you the actual VRAM consumption. I’m just re reusing this system as I already had it. By the way, you would need to install llama.cpp from their specific fork which you can find on their model card. So let me just download the model quickly. So I just need to install that hugging face

01:35 up CLI as it’s a new VM I’m using. That’s done. So let me now download. That should work. There you go. So while it downloads, if this is the first time you’re hearing about turnary bonsai, let me explain in very simple words what exactly this means. So this diagram primarily breaks this model into what it’s made of and what that actually buys you. At the top is the base which is quen 3.8 a 27 billion

02:06 parameter model at normal precision. From there three things happen to it. First every weight in the model gets crushed down to just three values -1 0 or one. That’s the turnary part or three part. It’s the reason the whole thing fits in under 6 GB at 1.76 bits per weight instead of the usual 16. Second, the attention mechanism. This is, as we know, the part of the model

02:37 that decides what to pay attention to in the text runs mostly on a cheaper linear method instead of the expensive full method, which means faster generation without the usual attention bottleneck. Third, because that linear attention is so much lighter, the model can hold on to a 262,000 token context window, which is what makes it actually usable for long documents or entire code bases instead of just short chats. And that is

03:08 what we are going to test out too. So, if you put all three together and you get a model that’s over nine times smaller than full precision while claiming to keep 98% of the original’s benchmark score intact, that is really awesome and impressive. But we are not going to fall for the hype. We are going to actually test it out on real codebase and code generation and few other tests. This is Fad Miraza and I welcome you to the channel. If you want to help out the channel, please just go to the homepage

03:38 of channel and become a member by following this uh link because that is what keeps the lights on. And huge shout out to all of these members of the channel. Recently, I also have started this free weekly AI newsletter which you can subscribe at vahadhatza.substack.com. If you’re looking to rent a GPU on very good price, you can find the link to mass compute in video description with a discount coupon code. model is downloaded. Let me now serve it with Llama server and I have enabled

04:10 Shinja so that we will be able to use tool calls. The model is being served and you can see that I have increased the number of slots here. Let me show you the VM consumption. So consuming just touch over 12 gig of VRAM. It is still higher and it is higher because the model raw size is just over 7 gig. Plus it also reserves gave cache for four parallel request slot by default. So if you want to reduce it just go back reduce this number here plus um just reduce the

04:41 number of slot just put in d-p parallel one and it is going to be consuming around under 8 gig of vm easy and quantise version more contise would be even lower anyway. So now we know what our system spec look like. Let’s now start testing it out. I have also configured this model with Hermes agent. For the first test, I’m going to give it this very interesting application which is called a Sereniti watch, a live wildlife migration monitor

05:13 for East African parks. Rangers log sightings on the ground. And this dashboard pulls it together. total sightings, animal counted, a 14-day trend, a live 24-hour activity gauge, migration pressure alerts, species breakdown, and a park leaderboard. And if you look at the back end, this is not a toy application. This is an actual application, which is Docker based, Postgress, fast API, and this dashboard which is auto refreshing wired together

05:43 with Docker Compose. I have already run it. It has got a real database which is running. Everything loads clean. No errors anywhere as you can see here. Right? But there is a one huge but very very subtle bug in this application and it’s a nasty kind. It’s in the last 24 or gauge as you can see on your screen. The back end computes its cutff using local server time instead of UTC while every sighting is stored in UTC. The containers runs

06:14 three or ahead of UTC. So the cutoff silently drops the most recent 3 hours of real sightings. The gauge and migration alert are under reporting right now which shows moderate and you can see that what effect would that be on tourists or even the park management overall or maybe for the animal and human safety. So, let me take you back and I’m just going to go to Hermes agent and give it this goal to fix this error because it’s a big

06:45 goal. So, I’m just going to paste it here to show you what I’m asking it to do. So, I’m asking it to just go through this application, find a bug, and fix it. I’m not telling it what the bug is, how to fix it, and um I’m just leaving everything to this thing. So let’s see how it goes about it. Because if it is a 90% quality of the original Quen, it should be able to find this bug and fix it as soon as possible. While that runs

07:16 through, I just wanted to show you one more thing. This diagram, don’t be scared. Let me explain it very simply. This diagram is primarily showing you the secret sauce as what what’s actually happening inside the model to make it that small. At the top you have got this 27 billion parameters split into three chunks. The main language bo backbone, the embedding and output layer and a separate vision tower for images. All of it except the vision tower gets turnary

07:46 compressed down to these three values as we talked about. Before the weights get crushed down to - 1 0 or 1, they are rotated first mathematically shuffled using something called a header rotation. um which is something like reshuffling a deck before you deal. So no single number is doing too much work. That rotation is baked in permanently. It cost zero extra storage. Then at infant’s time, the exact same rotation gets applied to the live data flowing

08:16 through the model. So everything lines up correctly. Now why does that matter? Without that rotation trick, squeezing weights down this hard usually breaks the model. This is a step that keeps it accurate and running all this needs special custom kernels which doesn’t work with stock lama.cpp CPP that is why I have rebuilt it um from the version they have shared in their model card they have also shared a very very finely written white paper if you’re interested

08:48 read it otherwise this diagram is sufficient enough for us mere models who don’t have PhD in machine learning and just like original coin it is also thinking a lot but you see it has already um found the bug and very very goes to solving it. It is still running. I’m just showing you the thinking or overthinking. It has already done a lot. Anyways, it is still running. So, let’s wait. And the model has come back after more

09:19 than an hour. Same amount of time which the original takes. But here is the fun part. Bonsai actually found a real discrepancy in the 14-day trend chart and fixed it. But that wasn’t the bug we planted. The actual bug is in the 24hour gauge. And interestingly, the model’s own reasoning walked right up to it as I showed you earlier. Right up to it. Even did the time zone math and then talked itself out of it. That’s the fun part. Concluding it was fine when it wasn’t.

09:51 So, it missed the actual real bug, but just, you know, fixed a random stuff which was really not necessary. And if I just do a hard refresh on this one, you will still see that it for the last 24-hour gauge, it still shows the same thing. And even if I, you know, let me open another window and do the curl here. And I’m just going to, you know, run this docker compour exit. So you see it shows the real settings 25 and recent

10:23 animals 2430. So end result, the bug was fixed but not the bug we planted. That still stays there. Let’s do another test. We will do the code generation this time. So for the code generation, I’m going to give it this prompt to create me a single HTML file which is going to simulate making a latte barista style. And there should be five or six steps in it like grinding coffee beans, temping the grounds, pulling an espresso shot,

10:53 steaming and frothing and then pouring the milk into cup and then a reset button to start the whole sequence again. So let’s see what it creates. It is also going to show us token per second and all that information. So let’s wait for u this to come back. You can see already looks pretty good in terms of token per second. So let’s see how long does it think and come back. Meanwhile, let me give you a very quick overview of this benchmark. So and this benchmark is from their own

11:23 website. So treat every numbers as their claim, not our verified result. The story it’s telling. By the way, across six categories, agentic tool use, coding, instruction following, reasoning, math, and vision. This compressed model stays close to the fulls size coin 3.8 8 it’s based on and even beats the newer coin 3 um the new version I forgot the name on the overall average but again um coin 3.6 Six is the older one. As you can see, the gap

11:53 between compressed and full is small across support. That is uh bit interesting for me. Nothing collapses in any one category. Whether that holds up matches what we actually saw in our own test is a separate question. This table is just their claim going in. I really don’t give much weightage to this table. And after a long time, it has created this file. And this is end result like 76.6. 63 and 14 minutes 21 seconds. Let

12:24 me open it in front of you. So, I’m just going to drag and drop. I already saved it here. The interface looks good. Not bad. All the steps are here. Let me start the sequence. It says press reset to begin the pore. Where is the reset? And why should I press the reset when I’m doing this? I’m just clicking here and I don’t see any reset button whatsoever. Here it says

12:55 start sequence. Nothing else. Sound is on to. So I’ll just go back. So this is a miss because the original coin never does that. I’ll just tell tell it that um nothing happens when I click on dot sequence button. So let’s wait for it to fix it and then we will test again. It says that it has

13:26 fixed it. Let me hard refresh and let me start sequence again till nothing happens. Okay. So one last try and I will go back and tell it to fix it. It is working again. So let’s wait. Okay, final time it says that it has caught the bug and also another typo. So I already have copied it as you saw. I’m just going to replace this whole code here and not the script. Let me copy the

13:58 whole thing. Saved. And let me now reload it. Refresh sequence. No, still nothing happens. So you see this one is a fail. Let’s do a vision test quickly. So I am going to give it this uh image of course a generated one where this man is mowing the lawn and something is happening on this trampoline supposedly. I’m asking it to describe exactly what is happening and then also write a short over-the-top neighborhood homeowner

14:30 association complaint letter from someone who witnessed this. So let’s see how model interprets this and thankfully med model came back fairly quickly. So this is a playbyplay where it is telling us what happened there. If I quickly read through it, it is saying that we have a seasoned veteran of the lawn, a growing veteran in Kaki polo and is that kakipo? Sorry, not this one. This one. Yeah, sort of I would say. And then you

15:01 know in the other you know on the right it has also described this you know amazing scene trampolines safety net and then um I would say it is okay if not world class. The actual coin is much better on this one but other than that it is not bad. Look read through this. drops back down onto that blue pad with a grace and joy of an Olympic gymnast. So that is good. And

15:32 now this is a complaint that to whom it may concern unsanctioned lawn warfare and aeriallessness. So the format is right and then timing is there. um unlicensed aerial acrobatics that is basically bit of a touch on human respectfully and with a trembling hand. This is a nice touch. Impressive. And you see this one is also quite nice. So this is quite a good response. I

16:03 think this is a pass with flying colors. Let’s finally do a multilinguality. So I’m going to ask the model to translate this sentence, spend less than what you earn and invest the difference in all of these languages. And of course as usual I will request you once the model comes back check out the response. And so far by just looking at the reasoning I think this is broken. You can see it is stuck at uh Tamil here and few other languages.

16:35 And the reasoning is also just stuck. Anyway, I will let it run a bit and see if it is able to proceed further. So far, it is just hallucinating. You see, it is just going in this line. It’s all broken. So, nothing is going to happen here. So, look, I think um this model is worse than the previous version. The previous version was actually quite impressive, which I showed you earlier, but this one really I don’t think so there’s anything to write home about. Um, so you just saw the model without

17:08 any hype or fluff. Let me know your thoughts in the comments.