DeepSeek-V4.1-Flash-1017 GSQ+RCO GGUF: Run Under 12GB VRAM Locally

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: DeepSeek-V4.1-Flash-1017 GSQ+RCO GGUF: Run Under 12GB VRAM Locally · Channel: Fahd Mirza · 08:21 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI models, Local and open-source models

Summary

The video demonstrates a locally run, heavily quantized DeepSeek V4.1 Flash model, which the presenter says runs with under 12 GB of VRAM and can generate a single-file HTML animation. It introduces the model and its GGUF builds, then explains how RCO allocates bits across model components and how GSQ improves the rounding of individual weights. The presenter also shows a multilingual naming test and says the responses look strong, while noting some misses in low-resource languages. The video closes by pointing viewers to further resources and inviting questions and feedback.

Key points

The presenter demonstrates an animated whirling dervish scene generated from a prompt as a single HTML file, with the model running locally.
The presenter says the model runs under 12 GB of VRAM using llama.cpp.
The video describes DeepSeek V4.1 Flash as a mixture-of-experts model and presents quantized GGUF builds at roughly 1.75, 2, and 2.25 bits per weight.
Quantization reduces the number of bits used to store weights, shrinking the model while introducing some loss of accuracy.
RCO allocates a total size budget across model components, assigning more bits to sensitive parts and fewer to tolerant ones.
GSQ starts from a GPTQ guess and learns how to choose among quantized values when rounding individual weights.
The presenter summarizes the roles as RCO deciding where to spend bits and GSQ deciding how to round each weight.
A multilingual test asks the model to create names for an animal-documentary TV channel in more than 80 languages; the presenter reports some misses in low-resource languages.
The presenter points viewers to the team’s GitHub repository and invites comments, questions, and feedback.

Chapters

Start	Chapter	Summary
00:01	Local demo and model setup	The presenter shows a generated whirling dervish animation, says the model runs locally under 12 GB of VRAM, and introduces the quantized DeepSeek V4.1 Flash build.
02:36	Quantization and RCO	The video explains quantization and how RCO allocates a size budget by giving more precision to sensitive parts of the model.
05:12	GSQ and how the methods work together	The presenter explains GSQ’s approach to rounding weights and describes how it works with RCO to create a GGUF file.
07:20	Multilingual test and closing	The presenter discusses the multilingual naming results, acknowledges some low-resource language misses, and shares ways to learn more or support the channel.

Quotes

The model is running locally — 00:31

GGIF is simply the file format that holds these compressed numbers. — 03:38

RCO decides where to spend the bits and GSQ decides how to round each one. — 05:45

Nothing is perfect, but again, it has done quite good. — 06:49

Concepts and entities

DeepSeek V4.1 Flash, GSQ, RCO, GGUF, llama.cpp, Fahd Mirza

Full transcript (original language)

00:01 This is Sema ceremony of whirling dervish. It starts out turning slowly, then speeds up, and the white skirt flares out into a wide disk. There is this tall hat, one palm up and one palm down, ripples running through the fabric, and a soft shadow on a wooden floor. All of it is a single HTML file drawn on a canvas with no libraries whatsoever. And I didn’t use any

00:31 external tool here. You can see that how cool that looks. You can even pause it. I created it just from this single prompt on your screen. The model is running locally, and that model is this DeepSeek, as you can see here with 763 billion parameters, but running locally under 12 GB of VRAM. I can also show you that. This is my VRAM consumption at the moment, and I’m

01:02 running it with llama.cpp. If you want to run it like I’m doing, these are all the commands which you need to follow to install starta, and then launch it with starta. I already have covered the detail in my another video. Coming back to this model, this is DeepSeek V4.1 flash, but in a quantized version coming from this external lab in Austria. We already have covered various models from this lab.

01:33 The original is a mixture of expert model with about 1763 billion parameters. Every layer has 384 experts, but only six of them are used for each token, and the context window is a million tokens. Most of the size sits inside these expert, which is exactly this is so suitable for compression. This team released three bits at roughly 1.75, 2, and 2.25 bits per weight all as

02:05 standard GGIF file, which you can also check out in their files section. Now, why I made this video once, you know, you can run it under 12 GB like I’m running. Plus, this benchmark table is so so good. You can see that Emmy 25 GPQ and Diamond and Life Code bench that two-bit build matches the original model score. And the real question is how you squeeze a model this big down and lose so little

02:36 as you just saw in this Seema the wish, which is, I think, is the best best stuff I have seen in a long long time generated by a model. So, the answer to that question is how did that happen is that two methods working together, GSQ and RCO. Let me explain them in as simple words as possible. Have a look at this image. So, this is the simplest picture here. A model is made up of billions of numbers

03:08 called weights. In a typical model, each one is stored with 16 bits. That gives you high accuracy but a huge file. Quantization stores each number with far fewer bits and here we are talking about two to four. The file gets much smaller and the price is some lost accuracy, right? Because every number is now an approximation. Deep 6 own releases already smaller than 16-bit with 8-bit for the dense parts and the four bits for experts and these

03:38 GGIF push it down to around two bits. GGIF is simply the file format that holds these compressed numbers. Now, that out of the way, let’s have a look at RCO. There is a catch with this squeezing which I just described. If you give every weight the same number of bits, you are treating every part of the model as equally important and it isn’t. Some layers are sensitive and a small error there hurts the output badly. Others are tolerant and you can crush them hard and

04:10 nobody notices. So the smarter move is to set a total size budget first and then decide who gets what. Sensitive parts get more bits, tolerant parts get fewer and everything adds up to exactly the target size. That’s the idea behind RCO. In this model, the dense part chooses from higher precision types while the expert choose from types between roughly one and three bit. The really good part is that RCO hits the budget exactly and it makes its

04:40 choices by looking at the model’s real loss not by guessing. And then have a quick look at this GSQ. So if you zoom into only single weight, once you have decided how many bits it gets, you still have to round it to one of a few allowed values. In this picture, the options are -2, -1, 0 and 1 which is what a two-bit grid looks like. The simple way is to snap to the nearest one but that’s not always the best choice for the model.

05:12 GSQ does something way smarter. It starts from a solid GPTQ guess then keeps all the options open at once scoring each one by how well it fits. As it trains, it slowly sharpens that guess like turning a dial until one option wins. That is the Gumbel Softmax idea in the name or GSQ. So this is a way to make a hard choice learnable, right? The result is accuracy. Now how they work together.

05:45 RCO works on the whole model and spends the size budget across the layers so bits go where they are needed. GSQ works on a single weight and makes sure it’s rounded well. RCO decides where to spend the bits and GSQ decides how to round each one. This is the, you know, easiest and simplest way you can understand that. So, GSQ first quantizes every tensor at every candidate type to a build a database options, and then RCO searches that database and picks one

06:16 type per tensor to hit the budget, and finally the pick gets stitched into one standard GGUE file, which I just showed you earlier here in this model card. Now, you already have seen this model in action where it created this schema and how good that looks. Let me show you another example where what I have done is I have done this multilingual test where I asked it to generate the best possible name for a TV channel focused on animal

06:49 documentaries in each and every language, and then I gave it this 80+ languages. Now, I have already checked it. This is one world-class response without any loss, without any hallucination, but also check your language and let me know in the comments what do you think that the name of channel is correct or not. The ones which I have checked, they look really good. Yes, there are slight misses here and there for the low-resource language. Nothing is perfect, but again, it has done quite good. So, I’m just going to scroll down,

07:20 and you can see that even for the regional languages, it doesn’t really look that it has gone with simple translation. It has tried the nuance and culture there too. So, let me know in the comments what do you think. If you’re interested in learning more about the mechanics and inners of these GSQ and um RCO, you can check out their GitHub repo or just search my channel. We have done heaps of videos around there. I will be doing another video where we will be doing more tests around it um

07:51 from various domains as we always do, but the purpose of this video was just to explain these concepts in bit more detail because there’s a lot of confusion around them. So, hopefully, you were able to understand this in as simple words as possible. And if you have got the question still, just let me know in the comments and happy to answer those. Also, recently I have started this free weekly AI newsletter, which you can subscribe at fahadmiza.substack.com. If you want to help out the channel,

08:21 please become a member. Thank you for all the support.