Qwen3.8-27B Runs on an 8GB Graphics Card: My Exact Free Setup

Author: Hyperautomation Labs · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Qwen3.8-27B Runs on an 8GB Graphics Card: My Exact Free Setup · Channel: Hyperautomation Labs · 06:37 · Watch on YouTube

Topics: Software engineering with AI, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

The video demonstrates running Qwen 3.8 27B on an 8GB graphics card using llama.cpp, comparing 2-bit, 3-bit, and 4-bit model files. The 2-bit version is fastest at 33 tokens per second, while the 3-bit version scores best on the coding test and runs at about 6.5 tokens per second. The presenter explains how to fit the model by adjusting GPU offload, context size, and chat-memory settings, and shows it working with a coding agent. Alternatives include free cloud GPUs and a Mac with 24GB of memory.

Key points

The test machine uses an RTX 5060 with 8GB of graphics memory, a Ryzen 5, and DDR5 RAM.
The model’s approximate sizes are 16.5GB at 4-bit, 10GB at 3-bit, and just over 7GB at 2-bit.
The 2-bit NGL 99 recipe runs at 33 tokens per second, using about 7.5GB of VRAM and 1.5GB of regular RAM.
The 3-bit ISTA recipe runs at about 6.5 tokens per second, uses around 5GB of regular RAM, and offers twice the context of the 2-bit recipe.
The 4-bit recipe runs at about 3 tokens per second and uses almost 11GB of regular RAM.
In the coding test, the 3-bit model scored 29/30, the 2-bit model scored 27, and the 4-bit model scored 22; all three found the bug and document code.
The presenter says NP1 limits the server to one conversation, while CTK and CTVQ8 reduce chat-memory storage from 16-bit to 8-bit to help avoid out-of-memory errors.
The model can be connected to tools such as Open Code through llama server, and the video shows it fixing a failing test.
The video suggests Kaggle’s free 16GB GPU as a cloud option and reports 12.7 tokens per second on a MacBook with 24GB of memory.
Troubleshooting suggestions include using a smaller model or context, lowering reasoning effort, specifying CPU cores, and closing Chrome to free graphics memory.

Chapters

Start	Chapter	Summary
00:00	Running a 27B model on an 8GB card	The presenter introduces the 8GB setup, the rented test PC, and the model’s memory requirements.
01:30	Setup and three model recipes	The video explains the llama.cpp setup and compares the speed, memory use, and context of the 2-bit, 3-bit, and 4-bit files.
03:02	Benchmark results and memory settings	The presenter compares coding and retrieval tests, then explains settings that prevent the 2-bit recipe from running out of memory.
04:34	Using the model and other ways to run it	The video demonstrates using the local server with Open Code and describes cloud GPU and Mac alternatives.
05:35	Troubleshooting and recommendations	The presenter lists fixes for common performance problems and recommends the 2-bit recipe for speed or the 3-bit recipe for stronger answers.

Quotes

That’s faster than you can read. — 00:00

The result, 33 tokens a second — 02:02

bigger wasn’t smarter, it was just slower. — 03:34

With those two changes, it fits with room to spare. — 04:04

Want speed? Pick A. Want the best answers? Pick B — 06:06

Concepts and entities

Qwen3.8 27B, llama.cpp, quantização, NGL 99, Research Lab ISTA, RTX 5060, OpenCode, Kaggle, Ollama, OpenAI

Full transcript (original language)

00:00 You have an 8-gig graphics card. Every guide for Quen 3.8 27B says you need 24. This is that model on an 8-gig card writing working code at 33 tokens a second. That’s faster than you can read. No subscription, no API key, nothing leaves the machine. In the next 10 minutes, you get the exact setup. Three versions depending on what you care about, the two settings that make it fit, and which version is actually the

00:30 smartest. The answer surprised me. I don’t own an 8-gig card, so I rented a normal gaming PC with one. An RTX 5060, the cheapest new Nvidia card, a Ryzen 5, and regular DDR5 memory. It cost 20 cents an hour. That’s the kind of PC a lot of you already have. Quick context. Quen 3.8 27B came out on August 5th. It’s free. The license lets you use it for work, and it’s been downloaded almost 7 million times in 2

01:00 months. People use it as a coding assistant, a private chat GPT, and the brain inside their automations. The catch has always been memory. Here’s all the math you need. The full model is about 54 gigs. A quant is a shrunk copy that stores each number with fewer bits. At 4-bit, it’s 16 and 1/2 gigs. At 3-bit, about 10. At 2-bit, just over seven. And this model has a lucky design. It has 64 layers, but only 16 of them keep

01:30 a growing memory of the conversation. So, the chat itself costs very little. The rule is simple. Whatever doesn’t fit on the card spills into your normal RAM, and the more that spills, the slower it gets. Setup is [music] two steps. First, grab llama.cpp. It’s the free engine underneath most local AI apps, and there’s a ready-made download for Windows and Linux. Second, download one model file. Which file depends on the recipe you pick. Recipe A is the two-bit file. NGL 99

02:02 puts every layer on the graphics card. The result, 33 tokens a second using 7 and 1/2 of the 8 gigs and barely a gig and a half of normal RAM. The context is 8,000 tokens, which is plenty for chat and small coding jobs. Now, three of those flags are doing the heavy lifting and I’ll show you why in a minute. Recipe B is a three-bit file made by Research Lab [music] ISTA, built to stay smart at small sizes. It’s 10 gigs, so

02:32 it doesn’t fit on the card. You don’t have to work out the split yourself. Fit on tells Llama CPP to put as much as it can on the card and the rest in RAM. It used 5 gigs of normal RAM, so a 16 gig PC is fine. Speed drops to 6 and 1/2 tokens a second, about reading speed, and you get double the context. Recipe C is the four-bit file, the one most guides tell you to get. Most of it lives in RAM, almost 11 gigs, so you want a 16 gig PC at minimum and 32 to be

03:02 comfortable. Speed is three tokens a second, fine for jobs you start and walk away from. But is it smarter? That’s the real question. Speed means nothing if the answers are wrong. So I gave all three the same jobs. Write a function that passes 10 tests, three tries each, find a bug hidden in a 170-line file, and pull one code out of the first page of a long document. All three found the bug and all three found the code. On the coding test, the three-bit file scored 29 out of 30. The two-bit file,

03:34 27, and the big four-bit file, the one everyone recommends, scored 22. On one try, it wrote code that crashed on every input. So on an 8 gig card, bigger wasn’t smarter, it was just slower. Here’s the trap I actually hit. Run recipe A with the default settings and it crashes out of memory. Two reasons. By default, the server sets aside room for four conversations at once, NP1 makes it one. And by default,

04:04 the chat memory is stored at 16-bit. CTK and CTVQ8 store it at 8-bit of the space. With those two changes, it fits with room to spare. You’ll also see people warn that going down to 4-bit chat memory makes the model quietly dumber. I tested that. Same prompt, same seed. Both versions got the right answers word for word on the bug hunt, and 4-bit saved a gig and a half of RAM. So, if you’re short on memory, it’s worth trying.

04:34 Every command in this video, plus the right recipe for 12, 16, and 24 gig cards is in a free PDF. Comment 8 gig and I’ll send it to you. Running it in a terminal is nice. Putting it to work is better. Llama server gives you the same kind of web address Open AI does, but on your own machine. So, any AI tool that lets you change the address can use it. Here it is inside Open Code, a free coding agent. About a dozen lines of config. This little shop project has a failing test. I ask it to find the bug and fix

05:05 it. It reads the code, spots that the discount check says greater than 10 when the rule says 10 or more, changes one character, and runs the tests again. All four pass. All of that on the 8 gig card. No graphics card at all. Two options. One, free cloud GPUs. Kaggle gives you a free 16 gig graphics card for hours every week. And the same recipe B file runs there through O Llama. Eight tokens a second, no setup.

05:35 Two, if you have a Mac with 24 gigs, it runs on the chip’s built-in graphics. On my MacBook, it does 12.7 tokens a second with the internet blocked. >> [music] >> Five things that go wrong and the fix. Out of memory. Add NP1 and the Q8 chat memory flags or switch to fit on. Painfully slow. Too much is spilling to RAM. So, pick a smaller file or a shorter context. Thinks forever before answering. Set reasoning effort to low.

06:06 Slow on a big server CPU. Tell it how many cores to use with T. And your browser uses graphics memory, too. So, close Chrome before you load it. Real talk. 90% of you watching haven’t subscribed. On Facebook, subscribe is the supporter button. That is what keeps the lights on here. On YouTube, it’s free and it tells the algorithm to send you more. Subscribe to support us. So, the short version. Want speed? Pick A. Want the best answers? Pick B, the

06:37 three-bit file, which beat the big one. Skip the biggest file unless you have lots of RAM and patience. An 8 gig card is enough to run one of the best free models out right now. Comment 8 gig for the full recipe sheet and tell me your card and your speed. Follow along on Instagram and Facebook for bite-size AI and the deep dive playbooks live here on YouTube. Subscribe so you don’t miss one.