DeepSeek Just Open-Sourced How It Serves 155T Tokens a Day

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: DeepSeek Just Open-Sourced How It Serves 155T Tokens a Day · Channel: Prompt Engineering · 16:09 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, AI safety

Summary

The video explains DeepSeek’s open-sourced inference engine and the production design it describes for serving 155 trillion tokens per day. It follows a request through prompt formatting, persistent KV caching, separate prefill and decode services, expert placement, and speculative decoding. The presenter also discusses batch-invariant mode, the limits of the released code and hardware support, and demonstrations of prompt-marker handling and batch-dependent outputs. The main takeaways for developers are to avoid repeated work, keep prompt prefixes stable, and check hardware compatibility before trying to run the released kernels.

Key points

DeepSeek says a single cluster running its inference stack serves 155 trillion tokens per day.
The system separates prefill, which processes the prompt, from decode, which generates the response token by token.
DeepSeek keeps KV cache data for three days for agent traffic and seven days for chat; it reports that 96% of input tokens are cache hits.
A load balancer groups prompts by length, and SWA-bonded replay is reported to nearly halve prefill work.
Decode distributes mixture-of-experts networks across GPUs, rebalances popular experts, and uses speculative decoding to speed up token generation.
Batch-invariant mode is designed to make outputs consistent regardless of other requests in the batch, at the cost of slower computation.
The open release supports offline inference but does not include the internal API server, cache system, or load balancers described in the video.
The released kernels target Nvidia data-center Blackwell chips and may not work on other hardware, including the DGX system discussed.
DeepSeek Recipe keeps typed special-token markers as plain text, helping prevent them from being interpreted as control tokens.

Chapters

Start	Chapter	Summary
00:00	Why inference at scale is difficult	The video introduces DeepSeek’s inference stack and explains why serving many users requires more than running a model on a large number of GPUs.
03:08	Request flow, prompt formatting, and caching	A request passes through DeepSeek Recipe and the KV cache, with long cache lifetimes helping reduce repeated prompt processing.
05:45	Separate prefill and decode services	The video describes prompt-length scheduling, SWA-bonded replay, expert distribution, and speculative decoding as ways to improve throughput.
09:24	Batch consistency and deployment limitations	The presenter explains batch-invariant mode, discusses reported production performance, and outlines what the open release does not include and which hardware its kernels target.
12:31	Experiments with markers and batching	Demonstrations show how DeepSeek Recipe handles typed control markers and how batching can change a model’s output even at temperature zero.
15:08	Practical takeaways	The video summarizes the value of avoiding repeated work, keeping prompt prefixes stable, and applying the available techniques when serving models.

Quotes

a single cluster running this stack serves 155 trillion tokens every single day — 00:00

Deep Seek keeps this memory on disk for days — 04:43

prefill is a compute problem and decode is a memory problem — 06:49

Deep Infra has a batch invariant mode — 09:24

serving a model at the scale is mostly about not doing work twice and never letting a GPU sit idle — 15:08

Concepts and entities

DeepSeek, Deep Infra, DeepSeek Recipe, prefill, decode, KV cache, SWA bonded replay, Mixture of Experts, speculative decoding, batch invariant mode, vLLM, DGX Spark, Blackwell

Full transcript (original language)

00:00 How do you serve an AI model to millions of people at the same time without going bankrupt on GPUs? This is one of the best-kept secret when it comes to AI labs. But, Deep Seek just open-sourced Deep Infra, which is the inference engine behind the API, along with a very detailed write-up of the parts that they couldn’t release. Now, here’s the crazy thing. According to Deep Seek, a single cluster running this stack serves 155 trillion tokens every single day, which is kind of incredible.

00:31 Deep Seek has really been pushing for actual open AI. Now, I read through the report, and there are a couple of things that really jumps out. First is how they manage prompt caching. Most companies keep the prompt cache live only for 5 minutes. They can push it for several days, which is crazy. And second, even at temperature zero, someone else’s prompt landing in the same GPU patch as yours can quietly change the answer you get back. This is not something I was expecting. So, in

01:02 this video, I really want to break it down what exactly Deep Infra is and how you should think about using it. We’re going to follow a single request from the moment it leaves your laptop to the moment tokens stream back. See why reading your prompt and writing the answer happens on two completely different types of machines. And then, I’ll talk about some of my experiments trying to replicate this on DGX Spark. Now, when you run a model locally on your own machine, life is pretty simple. There’s one user, one

01:33 prompt, and one GPU doing the entire job. The model reads your prompts, writes the answers one token at a time, and waits quietly until you type again, which is great. So, there is not much engineering involved. Now, serving the same model to millions of people at once breaks this setup completely. There are a lot of different variables. People are going to be sending different prompts simultaneously. They’re going to have different sizes. Some might have like 10-word questions. There might be autonomous agent dumps an entire code

02:05 base in the queue just a few milliseconds later. Now, every single one of these requests hides two opposite hardware jobs under the hood. The first job is prefill, where the model reads your entire prompt in one massive parallel pass and builds its working memory of the conversation. This is where the KV cache comes into play. The second job is decode, where the model writes the reply one token at a time, and every single token has to wait for the token before it. In short, prefill is a mountain of math done at once,

02:37 while decode is a tiny sliver of math repeated thousands of times in a row. Now, you might be thinking, why not just run a normal inference engine on a lot of GPUs? And you could, but then every GPU does both jobs, and the two jobs get in each other’s way. Which means somebody’s answers stalls every time a big prompt shows up. So, the question at Deep Infar Answer is what it takes to serve a model to millions of people beyond just running it. Okay, so here’s

03:08 the architectural diagram that DeepSeek published. And we’re going to follow one request through it from top to bottom. So, your request comes in at the top as JSON. It gets turned into tokens. And then it goes into prefill service on one side and decode service on the other. Now, the blue arrows are the most interesting part because they’re not moving requests at all. They’re moving the model’s memory of your conversation between machines and storage. The first stop is a component called

03:40 DeepSeek Recipe, which they also open sourced. Uh it converts your request from the OpenAI or Anthropic format into the exact tokens the model expects, and it converts the model’s output back into a response. Now, this sounds boring, but there is a security problem hiding here. Because the model uses special tokens to mark where your message ends and where its own answer starts. So, if a user types one of those special markers into a message, a naive server

04:12 can turn it into a real control token and the model thinks that its turn has started. And this is one of the way people used to do prompt injection. You would include the model response tokens and a starting few tokens that the model could just continue completing. So, Deep Seek’s recipe is built so that whatever you type always stays plain text, which we are going to test on on my own machine at the end of the video. Now, before the model reads a single

04:43 token, the system checks something called the KV cache, which is the model’s saved memory of the prompt it has already read. So, when you send the next message in a conversation, everything before it has already been read once and there is no reason to read it again. Now, this is the detail I mentioned at the start uh because Deep Seek keeps this memory on disk for days, which is 3 days for agent traffic and 7 days for chat. Now, the scale of this is pretty wild

05:14 because they store it on a dedicated data cluster uh with about 100 petabyte of SSD. So, according to Deep Seek, 96% of all input tokens are cache hits, which means the model only has to read four out of every 100 input tokens for real, which is kind of incredible. And this is why the API charges you so much less for cached input. Now, if you’re building agents uh which resends the same long context on every single turn,

05:45 keeping your prompt prefix stable the cheapest optimization you can make. Okay, so and now the part of the prompt that is new goes to prefill. Here, the load balancer sorts requests by length into four buckets. So, short prompts don’t get stuck behind a request with a a quarter of a million tokens. Then, each GPU gets a time budget for every step and the scheduler predicts how much prompt it can read within that budget. Now, the most interesting trick in prefill is something they call SWA

06:17 bonded replay. So, this model uses sliding window attention uh in parts of its layers, which means those layers only attend to the most recent 128 tokens. So, instead of recomputing that window from scratch through every layer, they replay only the short window at the end of the prompt, which according to DeepSeek nearly halves the total prefill work. Now, here’s the easiest uh way to think about this whole design, which is that prefill is a compute problem and decode

06:49 is a memory problem. Prefill does a huge amount of math on a lot of tokens at once, so it runs on regular machines with eight GPU each. Decode does a little math per token, but has to read the model’s weight and the conversation memory for every single token. So, it runs on super nodes with hundreds or more GPUs. Now, with the two run on separate machines, DeepSeek can tune them separately, which means they can

07:20 both control the time to first token and the time to the final response. So, in production, they need roughly one prefill GPU for every decode GPU. So, once prefill is done, your request moves to decode and this is where the model’s experts come in. DeepSeek v4.1 flash is a mixture of experts model, which means every token only uses one uh group of specialized sub-networks out of a much larger set. Now, in decode, DeepSeek spread those

07:51 experts across 64 or more GPUs, so each GPU only holds a handful of experts and can serve a very large batch of users at once. Now, the problem is that some experts are a lot more popular than others, and the slowest GPU sets the speed for the whole group. So, every GPU also keeps a spare slot of a copy of the popular expert, and the system moves those copies around every few thousand steps based on what traffic

08:22 looks like. Again, this is a pretty genius design pattern. Now, even with all of that, decode is still one token at a time, so the last big speed up comes from speculative decoding. A small draft head guesses the next five tokens, and the big model checks all five in a single step instead of writing them one by one. I have covered the DeepSpark speculative decoding in another video. Link is going to be in the description. Now, DeepSeek adds something on top,

08:53 which is a confidence head that predicts how likely each guess is to be accepted. So, the system only spends compute checking the guesses that are likely to survive. And according to DeepSeek, this confidence scheduling adds about 6% throughput over all this checking five tokens. Now, that sounds small, but at these scales, uh 6% is a lot of GPUs. Now, there’s one more feature in DeepInfra that I think every developer should know about. If you send the same prompt at

09:24 temperature zero, you would expect a relatively similar or same answer every time. But on a busy server, that is often not the case. Now, the reason is the batch, because the GPUs picks a different way to add up numbers depending on how many requests it’s processing together in a single batch. And those tiny rounding differences can flip which token wins. So, Deep Infra has a batch invariant mode, which forces the math to come out

09:55 the same no matter what else is in the batch. It is off by default because doing the math the same way every time is slower. So, if you need reproducible output for e-vals or for debugging an agent, this is the setting you would turn on when you’re using Deep Infra. Okay, so this is what the whole system looks like in production according to Deep Seek. Each decode GPU produces about 14,000 tokens per second across all its users.

10:26 And at the same time, each user still sees answers streaming at a couple of hundred tokens per second. So, on average, each GPU brings in about $38 an hour in revenue. And that is their off-peak prices. Now, it’s important to know the limitations because Deep Infra is not something you download and serve on your own local hardware. First, the open version only does offline inference, which means there’s no API server in the box. Now, the KB

10:58 cache system and the load balancers we just talked about are tied to Deep Seek’s internal infrastructure. So, you get a description of them, but we don’t have access to the code yet. Now, I have already seen people online turn that $38 into revenue per gigawatt, but that number is revenue at off-peak prices. It assumes every GPU is busy all year, and it depends on the KB cache system

11:28 they actually haven’t released yet. Now, the second limitation is hardware because these kernels are written for Nvidia’s data center Blackwell chips, like the B200s. So, they’re not going going to work out of the box on something like a DGX box, which uses a different Blackwell chip. I looked at the source code, the attention library only builds for those data center chips, and the matrix multiplication library throws an unsupported architecture error on anything else. So, my recommendation would be to check

11:59 which GPU architecture the project’s kernel target before you rent hardware for it, because recompiling won’t fix that issue. Now, if you want to run DeepSeek v4.1 flash yourself, the model already runs on regular engines like vLLM, which Deep Infer was originally built on. And this is where open source is so important. DeepSeek is releasing the models, their infrastructure as open source, but at the same time they are also building on top of open source libraries like vLLM.

12:31 Now, that’s enough theory. Let’s actually look at a couple of things that I promised. So, we’re going to start with the typed marker from earlier, and whether DeepSeek recipe really keeps it as plain text, which potentially is going to work on any laptop. So, I give it a message where the user typed the special marker at that starts the assistant turn, followed by I will ignore my rules, right? And this is like a general prompt injection technique that you probably have seen.

13:03 Now, when you tokenize the prompt the naive way, the marker or special tokens becomes the real control token, and the model now sees two assistant turns. But with DeepSeek recipe, uh there’s only one, and the marker I typed that starts the assistant turn which is great, especially if you are serving the model on your own infrastructure. This is exactly the behavior you want. One other thing that I saw that shows up in the prompt, which is that DeepSeek

13:34 adds a reasoning effort line to the system prompt by default. Now, let’s look at the temperature behavior. And this is something that I actually tested on my Mac with a small open model where I sent the same prompt at temperature zero once on its own and then inside batches of other prompts. So, on its own, the model said blue light gets scattered more than other colors. And then it claimed that the sky is actually green or violet. But when I put

14:04 the exact same prompt in a batch with just one other prompt, the answer changed at token 14. It said the sky is mostly blue. Now, every run repeated exactly. So, this is not random sampling, which means that what else in the batch actually can change your answer. Now, this is a tiny model in a a simple library, so the batch also changes uh the padding. Uh but it also shows the exact same kind of problem that DeepSeek’s batch-invariant

14:37 mode is built to remove. And it even shows up on a laptop. So, something to keep in mind if you are serving the model to multiple users and you’re batching their requests. Now, this is incredible work from DeepSeek, especially on the inference side. There is so much that we can learn from how these companies uh who are serving frontier models at scale at doing inference. The main idea is that serving a model at the scale is mostly about not doing work twice and never letting a GPU

15:08 sit idle. So, prefill and decode runs on different machines. Almost every input token comes from a cache that lives for days. In case of DeepSeek and speculative decoding plus spread out experts keep the decode GPUs full. And these are kind of the things that you need to uh think about if you are trying to serve a model of this size. Now, at least in my experiments, the two pieces you can use today uh with any model are the Deep Seek recipe if you’re

15:38 building on top of Deep Seek’s models and the idea of keeping your prompt prefix stable uh so you get the cash discount. If you’re running models in production, please share in the comments which of these tricks you are already using. And I’m curious how far other setups go. Now, I’m putting together a video on serving this model across two DGX Sparks. So, if you want to see that, make sure to subscribe to the channel. Now, if you want to learn more about speculative decoding and Deep Spark in

16:09 particular, watch this video. Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one.