DeepSeek Cut Their Memory 437x in Two Years

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: DeepSeek Cut Their Memory 437x in Two Years · Channel: Prompt Engineering · 13:34 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, AI models

Summary

DeepSeek V4.1 Flash is presented as an architectural redesign focused on reducing KV-cache memory to 890 bytes per token, 437 times less than the figure cited for V4 Flash. The video explains four techniques behind the reduction: splitting encoder and decoder work, sharing attention memory, quantizing global memory to four bits, and rebuilding local cache rather than storing it on disk. It also covers speed features and reports strong results on some coding and developer-agent benchmarks, but weaker performance on expert scientific tasks and factual recall. The main takeaway is that memory efficiency and practical performance can matter as much as parameter count, while benchmark results depend on evaluation setup.

Key points

The video says KV-cache usage fell from 389 MB per token in DeepSeek’s first model to 890 bytes in V4.1 Flash.
Causal encoder-decoder architecture uses the bottom half of the layers to encode prompts and lets the decoder share keys and values.
Compressed sparse attention lets layers borrow or reuse memory and token selections instead of each layer maintaining its own.
Global memory nodes are quantized to four bits; the video says this helps bring the footprint to 890 bytes per token.
Local sliding-window state is replayed and rebuilt rather than saved to disk, which DeepSeek describes as an approximation.
Engram retrieves common phrases from a large lookup table, while DSpark uses speculative decoding to draft tokens for parallel checking.
The video reports strong results on some coding and developer-agent benchmarks, but a sharp drop on expert scientific-agent tasks.
The presenter warns that benchmark scores can vary across evaluation harnesses and that the cited numbers are vendor-reported.
The closing argument is that raw parameter scale is not the only meaningful measure of model capability.

Chapters

Start	Chapter	Summary

Quotes

That is 437 times smaller, which is kind of incredible. — 00:30

The decoder doesn’t generate its own nodes. — 04:12

Well, they said don’t write them at all. — 07:17

check the scaffolding before you believe any chart. — 12:32

The number defining this model is that 890 bytes per token of KV cache. — 13:34

Concepts and entities

DeepSeek V4.1 Flash, KV cache, Causal Encoder Decoder (CED), Compressed Sparse Attention 2 (CSA2), Engram, DSpark, DeepSeek V4 Pro, DeepSeek

Full transcript (original language)

00:00 So Deep Seek just dropped V4.1 Flash and it’s all about efficiency. It’s not an incremental update. It’s an architectural change which is going to be a huge deal. Now to understand this, we need to put things in perspective. Back in 23, their very first model burnt through 389 MGB just to hold a single token of your prompt in memory. By version 3.2, they got it down to 48 MB. Then V4 flash

00:30 dropped it to 3.5 mgabytes. But in V4.1 flash, it takes just 890 bytes. That is 437 times smaller, which is kind of incredible. So this is not just an incremental update. The entire model was redesigned around shrinking that one specific number. Now to get there, they made four radical pads. First, cut the model in half. Second, let layers borrow

01:01 memory instead of calculating their own. Third, crush the remaining nodes into four bits. And fourth, stopped saving local cache to disk altogether. Don’t worry if these things don’t make sense yet. I’m going to explain them in a bit. If you’re running long horizon autonomous agents or feeding in huge code bases, that number is actually what drives your infrastructure bill and cost. Let’s walk through all four pieces because there is one interesting blind

01:32 spot where this architecture completely breaks down. And I’m going to show you that in a minute. But first, let’s look at what actually is eating your memory. Language model reads your prompt. Every layer writes down a small note about each token. It’s called a key and value. So it never has to reread your context when generating the next word. That stack of nodes is KV cache and it only ever grows. At older rates, a single 1

02:05 million token context eats roughly 48 GB of pure VRAM before you even count the model weights. And that’s significant. This is the number that stops you from using the whole context window of a model. With this new model, Deep Seeks makes a very simple point. Attention already solves the raw math of long context. The real world developer hits in production is not compute anymore. It’s actually the raw bites and how far they have to travel across your

02:36 hardware. Okay. So, every API call that you make does two completely different jobs. The first one is prefill. So prefill is model ingesting your entire prompt in a single massive pass. Decode is it generating the response back to you one token at a time. Think about reading as a wholesale work and writing is essentially slow trickle token by token. That’s why these large language models are called next token predictors.

03:07 On top of that, this is a massive mixture of experts architecture, meaning it only wakes up a tiny fraction of its brain for every given token. Now, it has 552 billion parameters in total, but each layer only activates six dynamic experts out of 384, plus one shared expert that runs on every token. But here’s an insane architectural shift. It fires 8 billion parameters to read a token and 16 to

03:40 write one. So ingesting context literally costs half of the compute of generating [clears throat] it. Now this is not by accident. It falls directly out of the first majoral architectural change. Okay. So the first one is what they are calling causal encoder decoder or CED. In a typical 40 layer transformer, every single layer computes and stores its own keys and values. So 40 layers means 40 separate piles of

04:12 nodes for KV cache. DC literally cuts the model in half and the bottom 20 layers act as a encoder and the top 20 act as a decoder. The decoder doesn’t generate its own nodes. It projects one single shared set of keys and values out of the final encoder layer. So the entire half of the network reads from the single shared memory bank instead of allocating 20 more set of its own.

04:43 That’s exactly where the 8 billion versus 16 billion parameter splits comes from. You only run the bottom half of the stack during prompt injection. So reading is literally half the work. It’s a huge cut in memory requirement, but you still have 20 encoder layers writing nodes into VRAM, which brings us to change number two. This is what they’re calling compressed sparse attention to or CSA2. Remember, no layer reads your

05:14 entire prompt. A small indexing scorers scans ahead and pick only the few hundred tokens that actually matter. That means an attention layer has two jobs. Write new notes and pick which old note to read. CSA proves most layers don’t need to do either. A full layer does the whole job. It computes its own nodes and picks its own tokens. A reindex layer writes zero nodes. It borrows memory from a full layer before

05:46 it, but reruns the picker to look at different parts of the prompt. And then there is a reuse layer which simply borrows both. It copies the notes and the selection directly in the decoder that leaves just one layer writing for repicking and 15 straight up copying. Now there is just one catch. Every layer still keeps a tiny sliding window of recent tokens in full resolution. that

06:16 local window is kept separate per layer and none of this sharing touches it. Now keep that in mind because we’re going to come back to it in a bit. That leads into change three and four and these go straight after physical storage. Change number three is they quantize the global memory nodes down to four bits because 4bit has almost no dynamic range. Every 16 values share a high precision scale factor. So tiny numbers don’t round away

06:47 to zero. The idea is that if you stack 4bit precision on top of the layer sharing and the encoder split, the memory footprint is going to drop to almost 890 bytes per token which is a 75% reduction from the V4 flash. Now fourth, those local sliding windows we just mentioned, every layer keeps one. So saving an idle conversation out of the SSD means writing 40 separate chunks to

07:17 disk. What exactly was their fix? Well, they said don’t write them at all. When the user sends another message, simply replay the last window of tokens through the network and rebuild the local state on the fly. In this case, the persistent disk storage drops to an eight. Now, Deepseek is upfront that this replayed state is an approximation, not an exact match. They claim that the quality is loss is negligible because they train the model to handle replay noise. But

07:50 that’s their self-reported test. So, take it with a grain of salt like you will do it with every other model. Okay, we’re going to look at what happens if you put all of the stack together. So in this case half the layers are writing the rest are sharing or copying and everything is stored in 4bit and local window never is saved to the disk. Okay. So here’s a payoff curve from the deepseek paper. Plotting the computing needed to generate one tokens against the size of the context window. Model

08:22 generations scaled straight up. V 4.1 flash is almost completely flat. Now, if you stretch your prompt context 256 times from 4,000 tokens out to full million and the cost to spit out a token only takes up by about 25%. So, if you ever had an autonomous agent who running tools for 3 hours, you know, you’re not paying for raw intelligence.

08:53 you are paying to drag that massive chart history behind it on every single turn. This makes dragging contacts dirt cheap. Okay. Beyond memory savings, there are two speed modules under the hood. That explains why the token generation moves so fast. It can generate up to 300 tokens per second, which is kind of incredible. So first is engram a massive 196 billion parameter lookup table running alongside the core

09:24 model. Instead of burning uh GPU compute to figure out common engram phrases, it simply retrieves them from static memory. And because it acts as a sparse dictionary, those extra parameters sit in ordinary cheap system memory rather than hogging precious HBM. I have a video on the topic if you are interested. Second is DSpark which is their speculative decoding engine. It’s a tiny fast helper model drafts five

09:56 tokens ahead. Massive model checks all five at once in a single parallel pass. Watch this video if you want to learn more. Plus on the API side, reasoning effort is now exposed integer slider from 1 to 100 rather than generic low, medium, and high toggles. So the system engineering is undeniably brilliant. But the million-doll question is how smart is the actual model? Before looking at the benchmarks and scoreboards, understand how this was trained.

10:28 Deepseek openly admits there is zero algorithmic innovation in their post training. its standard SFT, RL and distillation. The secret source is entirely in their synthetic data pipeline and automated agent environments. And keep in mind these are vendor reported numbers running at 100% reasoning effort. Compare the base architecture first. V4.1 flash sits at 552 billion parameters with 8 to 16

10:59 billion active while V4 Pro sits at 1.6 6 trillion with 49 billion active parameters. On LU Pro, Flash scores 74.1 against Pros 73.5, which is dead even while using a third of the weights and a quarter of active compute. On developer agent benchmarks, it punches way above its weight. 90.6 on terminal bench 2.1 beats Opus 5 at 89.1.

11:29 It ties OPUS at 74.2. 200 on deep and hits an absurd 3,471 ELO on code force. But now look at where it actually collapses on terminal bench 4.0 which isn’t a simple bash script benchmark u but a gruding scienceheavy agent environment flash tanks to 31.2 against ous at 51.8. It also gets crushed the exact same way

12:00 on humanity’s last exam and program bench. Now, Deepseek admits it directly in the text. There is noticeable performance gap on expert level scientific agent tasks compared to true frontier scale models. You see the exact same pattern on simple QA. When you compress a model down to a third of its size, you keep the logical reasoning rules, but you lose the encyclopedic factual recall. And watch out for the uh evaluation setup on Deep Sweet.

12:32 Testing the exact same checkpoint across different harnesses cause the score to swing from 74.2 down to 65.5. So the takeaway is check the scaffolding before you believe any chart. Okay. So what exactly are some takeaways from here? It was pre-trained from scratch on 45 trillion tokens with a native spots attention scaling out to 1 million token window. Both the model weights and the inference implementation are completely

13:03 open under any MIT license. There’s no there is no standard Ginga uh chart template included. You have to use the Python reference encoder or compile the rest recipe also is existing V4 Pro API calls automatically route to V 4.1 flash infrastructure. I think they changed it in the last couple of days though. The big takeaway here is that the AI industry is finally moving past treating raw parameter scale as the only metric

13:34 that matters. The number defining this model is that 890 bytes per token of KV cache. By redesigning the lay stack, quantizing memory, and dropping redundant cache from disk, they built a lean architecture that outclasses giant models on encoding and developer workflows, even if the deep scientific knowledge still demands massive physical scale. Anyways, I hope you found this video useful. Thanks for watching and as always, I’ll see you in the next