Run Qwen 3.8 27B in 8GBs Saluki 27B

Author: Data Science in your pocket · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Run Qwen 3.8 27B in 8GBs Saluki 27B · Channel: Data Science in your pocket · 07:11 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, Local and open-source models

Summary

The video introduces Saluki 27B, a 7.89 GB, 2-bit quantized build of Qwen 3.8-27B designed to run with stock llama.cpp. Across the benchmarks discussed, it retains much of the original model’s performance and scores higher on some tool-calling tests, while trailing on tasks such as competition math. The video also explains how to run it, describes an optional vision add-on, and notes limitations including formatting slips and verbose reasoning. Its conclusion is that Saluki is a promising preview checkpoint for people building local agents.

Key points

Saluki 27B 1.0 is a quantized checkpoint of Qwen 3.8-27B, compressed into a 7.89 GB GGUF file.
The model is designed to run on a 16 GB laptop using standard llama.cpp, without custom software.
The video reports that Saluki retains about 96% of the full model’s average across nine benchmarks.
On 120 tool-calling tasks, Saluki scores 88, compared with 84 for the full-size Qwen model; the video cautions that the sample is modest and some difference may be run-to-run variation.
On 100 parallel tool-calling tasks, Saluki scores 42 with a strict checker and 55 with a forgiving parser; about a fifth of multi-call replies need lenient parsing.
On 50 SWE-bench Verified issues, Saluki fixes 30 compared with 33 for the full model.
Competition math is a relative weakness: the video says Saluki retains roughly 82–85% of the full model’s performance.
The recommended setup uses llama.cpp with the Qwen 3.8 chat template; thinking can be disabled for fast, direct tool calls.
An optional vision projector adds image input; vision is not included in the main text-only file.

Chapters

Start	Chapter	Summary
00:00	Introducing Saluki 27B	The video presents Saluki as a compact 2-bit Qwen 3.8-27B build and previews its claimed tool-calling strengths.
01:40	Quantization and tool-calling benchmarks	The video explains Saluki’s size and runtime, then compares its tool-calling and parallel-call results with other models.
03:14	Agent tasks and performance trade-offs	Results cover parallel tool calls, software issue fixes, browsing, instruction following, coding, and the model’s weaker competition-math performance.
04:54	Running Saluki and its limitations	The video gives suggested llama.cpp settings, describes the optional vision projector, and summarizes limitations such as formatting slips and verbosity.
06:30	Availability and verdict	The video credits the contributors, notes where Saluki is available, and concludes that it is worth trying for local agent projects.

Quotes

What if a 54GB model fit in under eight? — 00:00

The result keeps about 96% of the full model’s average across nine benchmarks — 01:40

About a fifth of multi-call replies need the lenient parser. — 03:14

Running it is one command — 04:54

brutal compression and agentic reliability aren’t mutually exclusive. — 07:11

Concepts and entities

Saluki 27B, Qwen3.8 27B, Underdog, llama.cpp, GGUF, Berkeley function calling leaderboard, SWE-bench Verified, Hugging Face, Conway Research

Full transcript (original language)

00:00 What if a 54GB model fit in under eight? Meet Saluki 27B, a 2-bit build of Qwen 3.8-27B from a team called Underdog that runs on a 16 gig laptop with stock llama.cpp, no custom software. And here’s the twist, at tool calling, the thing that turns a chat model into an agent, this shrunken model actually beats the full-size original. Let’s break down how they pulled it off, the benchmarks, and the honest trade-offs.

00:35 Saluki 27B 1.0 is a quantized checkpoint of Qwen 3.8-27B, the open-source model that reportedly outscores Claude Opus 4.6 Max on artificial analysis. Underdog compressed it into a single 7.89 GB GG UF file using an importance matrix recipe built on ISTA Desta’s labs 2-bit release. It’s Apache 2.0. It’s a preview checkpoint with full training details coming, and it passed 109,000 downloads

01:06 in its first month. Why is that impressive? Because 2-bit quantization is the deep end. Crush 27 billion parameters down to two bits each and things usually fall apart, and agentic tool calling is the first casualty. The model has to pick the right function, fill in the arguments correctly, and know when no tool fits. Most tiny builds can still chat. Far fewer can reliably drive an agent

01:40 loop. Underdog’s answer is careful quantization, an IQ2 mix recipe that spends bits where they matter most. The result keeps about 96% of the full model’s average across nine benchmarks while shrinking the file roughly seven times. That size is the whole point. It fits entirely on a 16 GB laptop GPU, loads in vanilla.llama.cpp,

02:10 and needs no special runtime. Competitors like Me are require xlama V3. Saluki is just a standard GGUF. The headline test is underdog bench. 120 tool calling tasks pulled from the Berkeley function calling leaderboard frozen before any model was tested. Every model run in its own best runtime. Saluki passes 88.

02:40 The full-size Qwen 3.8-27B 84. Other two-bit builds trail behind Me at 85, STA at 76, Ternary Bonsai 2 at 70. Underdog’s own main build tops the chart at 91. Fair caveat from the team itself. 120 tasks is a modest sample, and part of the lead over the original is run-to-run variation.

03:14 Parallel tool calls firing several functions at once are even harder. On 100 parallel tasks with the official strict checker, Saluki scores 42 against the full models 35, which often makes one call and stops. With a forgiving parser that recovers small formatting slips, Saluki reaches 55. That formatting fragility is real, though. About a fifth of multi-call replies need the lenient parser.

03:48 But, benchmarks like these are proxies. The closest thing to real agent work, 50 genuine GitHub issues from SWE-bench verified, explore the code, find the bug, fix it, pass the project’s own tests. Saluki fixes 30, the full model 33, and Saluki keeps 27 of those 33 fixes. In web browsing, clicking from a home page to an answer across 150 questions, Saluki answers 60 40%. For an 8 GB file, that’s most of the

04:21 original’s agentic muscle. Elsewhere, instruction following actually edges above the full model’s public number 93.5 on I feebles loose measure. Coding on MBP++ lands at 78. The honest weak spot is competition math. Aim scores around 80, roughly 82 to 85% of the full model. Underdog is up front about the positioning built for agents, not math

04:54 olympiads. Running it is one command, llama server, the GGUF file, ginger flag for the Qwen 3.8 chat template, flash attention on 32K context. You get an OpenAI compatible API on port 8080, drop-in for existing agent frameworks. Recommended settings, thinking on with temperature 0.6 for general use, for fast direct tool calls, disable thinking

05:25 and set temperature to zero. Thinking mode is on by default and switchable per request, same behavior as Qwen 3.8. Expect it to reason at length before answering. That’s where the quality comes from. And while the main file is text only, an optional vision projector 629 to 928 megabytes adds image input through the same OpenAI chat API.

05:56 Text plus pictures when you need them. The honest limitations, straight from the model card, it keeps only 82 to 85% of competition math. Letter level wordplay palindromes, vowel rules, alphabetical ordering is its least favored instruction type. About a fifth of parallel call replies have small formatting slips. With thinking on, it can be verbose. And remember, vision is an add-on, not

06:30 built in. Credits where due, the Qwen team for the base model, I stayed at S Lab for the 2-bit release it builds on, Berkeley for the function calling tasks, all Apache 2.0. It’s on Hugging Face under Conway Research with over 108,000 downloads last month, and it loads anywhere llama.cpp runs, LM Studio, Ollama-style apps, your own server. Saluki also ships inside Underdog’s own app alongside dozens of other models.

07:11 The verdict, Saluki 27B proves that brutal compression and agentic reliability aren’t mutually exclusive. 7.89 gigabytes, stock llama.cpp, tool calling that edges out the 54 gigabyte original. It’s a preview checkpoint, so expect iteration, but if you’re building local agents, this is the one to try this week. Like, subscribe, and tell me what you’re running it on. See you in the next one.