Underdog Saluki 27B 1.0 Locally: Beyond Qwen and GSQ+RCO

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Underdog Saluki 27B 1.0 Locally: Beyond Qwen and GSQ+RCO · Channel: Fahd Mirza · 08:42 · Watch on YouTube

Topics: AI evaluation and benchmarks, Software engineering with AI, Model inference and optimization, AI models, Local and open-source models

Summary

The video tests the Saluki 27B 1.0 model, a heavily compressed two-bit quant, and questions whether its reported average benchmark performance reflects how well it works in practice. On local tests, it performs impressively on creative writing, animated HTML generation, and multi-step physics reasoning, though the vision functionality does not work for the presenter. It gives a mostly plausible answer about Australian fixed-rate home loans but ends with a confident factual error. The presenter concludes that the model is interesting, but recommends sticking with the original Qwen model and its original quantizations.

Key points

The model card reports 96% of the full model’s performance across nine benchmarks, but the presenter notes that individual scores can differ substantially.
The presenter questions the tool-calling results because they come from a small test and a different evaluation setup than the full model’s public scores.
Saluki is described as being built on an existing two-bit quantization, but the method used to preserve tool calling is not explained.
In a creative prompt involving philosophers, the model produces responses the presenter finds inventive and entertaining.
The model generates a self-contained HTML page with an animated, rotating donor kebab, fire effects, an intensity slider, and a second tab of varieties from different countries.
The presenter says the vision projector does not work in their local setup, so they restrict testing to text prompts.
For an ISS question involving orbital speed, period, sunrises, and gravity, the model gives numbers and explanations the presenter says match their checks.
Its response about Australian home loans is mostly plausible, but it incorrectly claims that 15- and 25-year fixed loans are popular.
The presenter warns users to verify facts from two-bit models and says they would still choose the original Qwen 3.827B and its original quantizations.

Chapters

Start	Chapter	Summary
00:01	A cautious look at Saluki 27B	The presenter introduces the model, questions its benchmark claims, and compares it with previously covered Qwen models.
01:34	Local setup and creative test	After explaining the unclear tuning behind the two-bit quant, the presenter runs the model locally and evaluates its creative response.
03:13	Animated HTML and world knowledge	The model builds an animated donor kebab webpage, and the presenter checks its rendering and list of regional varieties.
05:21	Physics reasoning and factual reliability	The presenter reports strong results on an ISS reasoning question, then finds a confident factual error in the model’s answer about Australian home loans.
08:07	Conclusion and recommendation	The presenter reiterates the need to check facts and says they would stick with the original Qwen model and quantizations.

Quotes

I’m really not sure about this model. — 00:01

I think creativity is really good. — 03:13

This looks sublime. — 04:19

hallucinated with a confidence which is exactly why you should check a twobit models facts always. — 08:07

I would still stick with the original Quen 3.827 billion and the original quads from Quen. — 08:42

Concepts and entities

Saluki 27B 1.0, Qwen, GSQ, RCO, llama.cpp, Fahd Mirza, Hugging Face

Full transcript (original language)

00:01 I am really not sure about this model. I even don’t know why I am covering this. Maybe it’s a slow day in AI and I didn’t find any other model to cover. But today I’m covering this underdog Saluki 27 billion. But again, I want to be upfront. I’m not sure about this model. The card says it is 3.8 27 billion shrunk from 54 gig down to 7.89 89 GB with 96% of the full model’s

00:32 performance. But that 96% is an average across nine benchmarks and the individual numbers tell a very different story as you can see from their hugging face card. Just one example, AME 2025 drops from 96.7 to 79.2. The tool calling win comes from a small 120 task test and the public scores for the full model come from a very different harness. Now here is the thing

01:02 we already have covered the full coin 3.827 827 billion an amazing amazing model despite of its overthinking and then look at this we have also covered this Dashlab um GSQ and this RCC long time back where RCO sorry where um it just took another step with 23.827 827 billion. What these guys have done, they have taken it to another level. As far as the card tells us, some this model is

01:34 built on an existing twobit quant from this dashlab model. But how they tuned it to keep tool calling intact is not explained at all. And at two bits, I want to see whether it holds up beyond single shots or not. So let’s test it out for real. This is Fad Miraa and I welcome you to the channel. I already have served this model with vision on my local system with llama.cpp as you can see on your screen plus this is the vramm consumption at the moment on my

02:05 system. This is the server. Let’s first start it out with a creative test. Now this creative test is asking it let me just press enter here because sometime lama.cpp is just uh not play the ball. I’m asking it what is life and then few philosophers are giving their answers and then there is a twist there where Fahad Ma says it’s membership to my channel then I’m asking it what would these philosophers say if we ask it what is wife and then we have run the model

02:38 and the model is still thinking let’s wait for it response and we will check so look do those whiskey amazing amazing philosophers by the way it says it’s a companion in suffering Socrates It is partner and inquiry. It is total mind of the household nit power you must master. Very nice. And I’m not sure about fian um slip here. Shopopenhar. Okay, this is cool. Very nice. Patience in motion. I think this is the best one and it is your most

03:13 loyal subscriber. I think creativity is really good. remember two bit quant on top of the dashlab which is on top of 3.827 billion. So first test in my opinion I think at least it was worth the time to get it installed so far. Let’s do another coding test now. So this is our second test that you can see we are asking the model to build a self-contained HTML file with an animated donor kebab on fire and then another second tab listing donor

03:46 varieties from across the world. We are testing its coding and animation ability at two bits plus how much world knowledge this one keeps. Let’s wait for it to finish and we will check out the response. The model has generated the file. Let me show it to you in the browser. There you go. So, look this. There are two tabs as you can see. Donor on fire and donor around the world. We’ll get to the other one shortly. But look at this one. This is so good. The fire tab now has a rotating tapered donor with a heating

04:19 panel behind it. Flames are licking up both sides. Sparks, smoke, intensity slider is also there. Let me move it. Oh, wow. This looks sublime. What do you think? Let me decrease the intensity. This looks so cool. Very close to the original uh coin model. This is so good. This is so good. Even the rotation looks pretty good to me. Now, this is where it

04:50 has come up with a lot of names. Um and you can see that it has given us lot of countries, lot of varieties of donut kebab. So, please check it out. Let me know what do you think from your country if it is correct or not. If you cannot read it, let me make it bit bigger for you. Now check it out. Australia halal snack snack pack. I think this is correct. And this is so famous, so yummy here by the way. And uh from Germany, there are two of them like

05:21 from Berlin I guess and from Durham Donor. And there is third one too. So seems quite popular in Germany too. Turkey is also riddled with uh these donor kebabs. Yep, pretty good. So, please let me know your country. But again, this one is really good. I think very impressive rendering. Let’s do another test. Next up, let’s do a scientific reasoning question with a touch of math. By the way, I have tried images with this model

05:53 because supposedly the vision projector is there but it’s not working. So I have tried even in Python code in llama.cpp doesn’t work at all. So I’m just going to stick with the text questions at the moment. The prompt in front of you is asking the model to work out the ISS orbital speed period sun rises per day and gravity at altitude. Then explain why astronauts float. So we are testing whether it’s multi-step math and physics reasoning survived the twowbit squeeze

06:26 or not. Let’s wait for it and we will check it out. And the model has come back with the response. And you know what? It is again a very impressive answer especially in terms of the way it has thought. Uh I wish I could show you the thinking while it was going on. It is such a profound thinking broad deep and then verification at multiple levels. But the answer it got every number right speed period gravity and the freef fall explanation all matching

07:01 the calculations. I have rechecked it with um judge model. Simply amazing. Now this is really a good sign for a two- bit model. Um this is so good. Now let’s ask it a final question. I’m asking it. Why don’t Australian banks offer 30-year fixed rate home loans? So this is not only checking out its uh thinking but also if it knows about other regions and how fresh the data is and if it could infer from its training set as why is uh that the case.

07:37 Okay. And I think this time model has come back with a response but I guess struggling a bit reason uh the reasons which it has given are mostly sound like if you read through it lenders carry the interest rate risk funding it’s short-term and Australia has no Fenny and Freddy style secondary market like they have in US. Um but towards the end I think it has faulted because the ending is totally wrong. It says banks offer 15 and 25 year fixed loan as a

08:07 popular value play. Um which is totally wrong. They don’t do it. Seriously, the Australian bank are so rich uh but they are so miser and so uh tight hours I would say um I haven’t seen any fixed term maybe 2 years um maybe maybe 5 years by some lenders but not everyone. So I think it explained the why well but made up a confident detail at the end. So hallucinated with a confidence which is exactly why you should check a twobit models facts always. That’s it. Let me

08:42 know your thoughts about this model and again I would still stick with the original Quen 3.827 billion and the original quads from Quen. Thank you for all the support. Uh please subscribe to the channel. text.