Tencent Shrank a 1.5TB AI Model to 214GB — Here's the Trick

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Tencent Shrank a 1.5TB AI Model to 214GB — Here’s the Trick · Channel: Fahd Mirza · 08:17 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, AI models, Local and open-source models

Summary

The video explains Tencent’s Angel Slim toolkit and its Sherry quantization method, which the presenter says reduces the 770-billion-parameter I4 model from about 1.5 TB to around 214 GB. He describes the method as converting weights to -1, 0, or +1 and guaranteeing one zero in every group of four, allowing each group to be stored in five bits. The presenter says benchmarks barely changed, while noting that running the model requires substantial GPU resources. He also outlines that viewers can follow commands to download, run, benchmark, or re-quantize the model.

Key points

Angel Slim is described as Tencent’s toolkit for shrinking large AI models.
The video focuses on the I4 model, which the presenter identifies as having 770 billion parameters.
The model’s full-precision weights are said to occupy about 1.5 TB, while the quantized version is around 214 GB.
Sherry quantization maps weights to three values: -1, 0, or +1.
The method guarantees exactly one zero in every group of four weights.
According to the presenter, the resulting 32 possible patterns can be encoded in five bits, or 1.25 bits per weight.
The presenter says benchmarks barely changed after the size reduction.
Running the model requires a substantial multi-GPU setup; the presenter mentions four to ten GPUs or two to three H100s with 80 GB of VRAM.
The video’s closing walkthrough mentions cloning and building llama.cpp, downloading the quantized model, running prompts, benchmarking, and optionally re-quantizing.

Chapters

Start	Chapter	Summary
00:01	The headline and Angel Slim	The presenter introduces the claim that Tencent compressed a large AI model and explains his prior coverage of Angel Slim.
01:45	What Angel Slim does	He introduces Angel Slim as a toolkit for shrinking models and gives the I4 model’s original and compressed sizes.
03:58	Explaining Sherry quantization	After criticizing the documentation’s clarity, the presenter begins a simplified explanation of the quantization process.
05:06	Ternary weights and compact encoding	He explains mapping weights to three values and packing groups of four, each with one guaranteed zero, into five bits.
07:46	Running the model	The presenter outlines the hardware needs and the steps involving llama.cpp, downloading the model, prompting, benchmarking, and re-quantization.

Quotes

Angel Slim is Tencent’s toolkit for shrinking large models. — 01:45

Benchmarks barely moved. — 02:52

This is called as ternary, three values. — 05:41

Sherry forces exactly one zero into every group of four weights. — 06:12

Five bits, four weights. Do the division, and that’s one and a quarter bits per weight. — 06:43

Concepts and entities

Angel Slim, Sherry quantization, I4, Tencent, llama.cpp, Fahd Mirza

Full transcript (original language)

00:01 All right, so I woke up this morning and did something which I never like to do, that is to scroll through X. Now, X is full of slop, full of misleading information and fluff, but every now and then you find something which makes your day. At the moment, on your screen, you can see what is trending. Tencent shrinks 1.5 TB AI model to 214 GB. Seven times smaller, and the method behind it is called as Sherry, built into a tool

00:37 called as Angel Angel Slim. Now, I want you to look at that word, Angel Slim, because when I saw this headline, I didn’t feel surprised, I felt vindicated. Let me show you why. This is my channel, as you might have guessed by now, and this right here, this video, running a 2-bit AI model locally with Angel Slim, 5 months ago, 5 months, back when nobody was talking about it, no trending pages, no headlines, we were already hands-on

01:09 with this exact tool, this exact team at Tencent. So, when the whole internet discovers Angel Slim today and acts like it’s brand new, well, you and I already know it. That’s what we do here. We cover the thing before it’s cool every single day. Now, let me be actually very up front with you. This is not a hands-on video, but I will make it worth your while. And if by the end of this video, you’re unable to understand what exactly Tencent has done now, I will return your money back.

01:45 So, watch until the end of the video, and then we will discuss this money matter. So, let me first actually explain what this is, because the headline makes it sound like magic, and it’s not. It’s clever engineering, that’s about it. Angel Slim is Tencent’s toolkit for shrinking large models. That’s it. Think of it as a compression suite for AI. The tagline right there says it, more accessible, more comprehensive, more efficient. It bundles a whole set of tricks for making enormous model small

02:20 enough to actually run, and this is a proof of it. The recent I4 model, they have shrunk it, and that is what that headline is telling you. We already also have covered this I4 locally and also with API from various angles, and it is quite a good model. No doubt about that. Now, this model is 770 billion parameter. In its full form, the weights are about 1 and 1/2 terabytes. That is not a laptop. That’s not even one server. That’s a

02:52 rack. I can’t even run it on one GPU on my system. They ran it through this Angel Slim using a method called as Sherry or Sherry quantization and squeezed it down to around 214 GB. Same model, roughly the same smarts, a fraction of the size. Benchmarks barely moved. As I said, this is not a hands-on video. I’m not going to spin this up live because to run a 770 billion parameter model, you need a serious cluster, the kind

03:24 which we don’t have. But, here is what I’m going to do. I will walk you through the exact commands step by step, the full plan. So, if you do have that rig or you get access to one, you can follow along and run this yourself. And even if you don’t, you will understand exactly how it is done. But, if you want to check out the smaller version, 2-bit one, you can go to my channel, search with Angel Slim, and you should be able to follow this video. Now, I think one reason this method from Tencent hasn’t really caught up is due

03:58 to various things. The documentation is uh how should I put it politely? Rubbish. Uh I’m not sure if that’s a polite word. It’s not good. Look at this diagram from documentation. I mean, we can barely make out any sense out of it. Most of it is is is a jumble of English and Chinese. Uh so, very hard to understand even if you go through their page. If you read through this paper around Angel Slim, again, I think you need a PhD in machine learning in order to understand this concept and it is full

04:31 of uh this thing. I will come back to this diagram later, but as is uh the case with our channel, I’m going to first unpack what exactly this whole Sherry quantization and this whole thing which Tencent has done in as simple words as possible. So, let’s have a look at this diagram. And simplicity means that I will talk a bit longer than usual, so just allow me 2 to 3 minutes and just listen. Now, look at the top row. These are the model’s weights in full precision.

05:06 Numbers like 0.42, -1.7, 2.1. They are long, messy, precise decimals. And a model this size has hundreds of billions of them. That is why it’s 1 and 1/2 terabyte. It’s a storage cost of remembering all these exact numbers. Now, watch what happens in step two. Sherry looks at each of these messy numbers and rounds it down to just one of three choices: -1, 0, or +1. That’s it. And you guessed it right, ternary

05:41 bonsai if some of you are thinking about it. Now, positive, negative, or nothing is the go here. This is called as ternary, three values. So, -1.7 just becomes -1. .03, which is basically nothing, becomes zero. We have thrown away all that decimal detail, and it turns out the model barely misses it. And the clever part is really good. I really like it that this is a trick. Um >> [clears throat] >> Most of those other companies haven’t really gotten it right, but they it

06:12 seems they have. Look at the highlighted box. Sherry forces exactly one zero into every group of four weights. Always one. It doesn’t wait to see if a zero shows up naturally. It guarantees one. Zero means skip me, I don’t count. That’s it. And why does that matter? I mean, that’s a question now. Because once you know every group of four has exactly one zero, there is a fixed, limited number of patterns that group can possibly be.

06:43 Two, 32 of them, to be precise, by the way. And 32 patterns fit perfectly into five bits of storage. Five bits, four weights. Do the division, and that’s one and a quarter bits per weight. No waste, nothing left on the table. That’s the whole magic. Messy decimals become plus one, zero, minus one. Forces one zero per group of four. Packed it tight into five bits. And a one and a half terabyte monster

07:14 collapses down to 214 gigabytes, while still being almost as smart as the day it was ba- born. Now, look at this diagram. I don’t know how to explain this. This is so, so much um out of my league. I think this is for the smart people. So, we will just stick with our own explanation. Now, I hope that you got the whole concept right. Still, if you don’t understand what I just explained, just

07:46 let me know in the comments how much you have paid, and I will return your money back. Towards the end, these are all the command, the whole step-by-step action plan which you can run in order to get this done, but you would need around, you know, multiple GPUs like 4 to 10 in order to get this running, uh or maybe two or three H100s with 80 GB VRAM. Anyway, what is happening here is uh that we are just git cloning llama.cpp. We are building it from their branch,

08:17 and then it’s going to take a bit of a time download this quantized version of the model, and then you can run the prompt. You can also do benchmarking, and then if you like, you can simply just re-quantize it if you like. Recently, I also have started this free weekly AI newsletter which you can subscribe at fahadmirza.substack.com. Also, if you want to help out the channel, please become a member. Thank you for all the support.