Local AI 202: Get Massive Speed-ups with Speculative Decoding

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Local AI 202: Get Massive Speed-ups with Speculative Decoding · Channel: Hugging Face · 1:21:45 · Watch on YouTube

Topics: Model inference and optimization, Local and open-source models

Summary

This livestream introduces speculative decoding as a way to increase local LLM generation speed by using a smaller drafter to propose tokens that a larger target model verifies, while preserving the target model’s output distribution. The opening covers inference performance metrics, the prefill and decode stages, practical use through llama.cpp and Transformers, and key trade-offs such as memory use and drafter quality. Eldar presents Speculators and its training and deployment workflow, including data preparation, hidden-state extraction, training choices, and adaptive verification. Martin then explains Gemma 4’s multi-token prediction drafter architecture, how it reuses target-model computations, and why speed-ups vary by task and hardware.

Key points

Speculative decoding can improve decode throughput by letting a smaller drafter propose multiple tokens for a larger target model to verify.
The target model still controls the accepted output; speculative decoding is described as lossless in output distribution, although implementation and kernel differences can cause token-level divergence.
Prefill processes the prompt, while decode generates tokens one at a time and repeatedly reads model weights and the KV cache, making memory bandwidth a major bottleneck.
Drafters work best when they are well matched to the target model and its domain; poor matches or overly long drafts can waste computation.
llama.cpp, llama.app, and Transformers provide ways to use speculative decoding, with model repositories and model cards offering drafter details and setup guidance.
Speculators supports several approaches, including Eagle Tree, MTP, D-Flash, D-Spark, and D-Flash 2; these differ in how they predict future tokens.
Training a speculator involves generating target-model responses, preparing data, extracting or streaming hidden states, and choosing training and serving settings.
Eldar describes cross-distilling data from larger models, quantizing stored hidden states, using the Muon optimizer, and sliding-window attention as practical training techniques.
Adaptive verification can choose how many drafted tokens to verify for each request, balancing throughput and interactivity.
Gemma 4’s MTP drafter is a small model that reuses target-model activations and KV-cache information; gains tend to be stronger for structured tasks such as coding and mathematics than for creative generation.

Chapters

Start	Chapter	Summary
00:02	Introduction and livestream agenda	Marve introduces the session, local inference tools, and the guest presentations on Speculators and Gemma 4 MTP drafters.
04:12	Inference stages and performance metrics	The presenter contrasts generation with and without speculative decoding, then explains prefill, decode, throughput, and latency.
09:23	How speculative decoding works	A small drafter proposes several tokens for a target model to verify, with corrections made when a draft is rejected.
14:35	Choosing and using a drafter	The presenter discusses matching the drafter to the target model and domain, limiting draft length, and using available tools and model repositories.
20:59	Speculators overview	Eldar introduces Speculators, its Hugging Face-compatible format, supported algorithms, and focus on D-Spark and D-Flash 2.
26:41	Data preparation and cross-distillation	The training workflow starts with target-generated answers, and experiments suggest data from a larger family model can also train smaller-model speculators effectively.
33:24	Hidden-state extraction and storage	Eldar compares offline and online hidden-state extraction and describes quantization as a way to reduce the large storage burden.
37:30	Training choices and techniques	The talk covers speculator architectures, optimization choices such as Muon, and sliding-window attention for more stable long-context performance.
42:10	Serving and adaptive verification	After describing deployment and block-size tuning, Eldar explains adaptive verification and presents throughput and interactivity results.
48:54	Results, deployment guidance, and Q&A	Eldar discusses combining quantization with speculative decoding, recommends model cards and domain-specific training, and answers questions about output equivalence.
52:03	Gemma 4 MTP drafter architecture	Martin introduces the Gemma 4 drafter and explains how its compact architecture mirrors parts of the target model.
1:00:22	A round of MTP generation and verification	Martin walks through how the drafter uses target-model activations to propose tokens that Gemma 4 can verify in a single forward pass.
1:07:09	Speed-up trade-offs and efficient embeddings	The talk covers output consistency, task-dependent speed-ups, and clustering token embeddings to make the drafter’s language-model head more efficient.
1:15:26	Benchmarks, tuning, and closing questions	Martin discusses variation across hardware and workloads, suggests testing different draft lengths, and answers a question about cluster errors before the livestream closes.

Quotes

this is lossless, meaning you don’t compromise from the quality of the generated text — 04:12

So, the small model has to be good. Uh, so there’s no free lunch. — 14:35

The main feature of speculators library is that it implements a lot of these state-of-the-art algorithms for speculative decoding — 23:35

the principle still is is that the output should remain the same. — 1:07:09

But try it out. Try a different amount and see how it changes — 1:18:02

Concepts and entities

speculative decoding, llama.cpp, Speculators, vLLM, D-Flash, D-Spark, MTP, Gemma 4, Eldar Kurtic, Martin

Full transcript (original language)

00:02 Hello everyone and welcome to this live stream on speculative decoding. I in case you don’t know, I am Marve and I will be giving you a primer on speculative decoding. We have amazing guests. So without further ado, I want to share my screen first and then let’s go with this one. So we will first talk about

00:33 a bit like what speculative decoding is and I will give you a primer on llama.cpp and llama.app and llama.app is like a front end in front of llama.cpp. And then we have Eldar from Red Hat who is going to present about speculators and we will have Martin who is going to talk about Gemma 4 multi-token prediction drafters. So let’s go. Today when you say that you are running your models locally, everything is actually abstracted for you and this is

01:06 for instance for reference this is llama.app and it has like desktop apps for macOS and Windows and it is wrapping llama.cpp the OG inference engine for local inference. And it it handles everything for you. So if you were actually using llama.app, if a model has a speculative decoding drafter with it which gives you speed ups, it’s already doing that to give you extra speed ups

01:38 without you knowing it which is pretty cool. So for instance in here I’m serving Gemma 4 and Gemma Gemma 4 has a drafter so this is extra fast compared to what you would do with Gemma 4. So in here you see the MTP here. Um and like if you don’t know about it, please go check out llama.cpp and download the Windows or the Mac OS app and just start serving with few clicks. Um it also has model recommendations for

02:09 your computer. So like if you have like a MacBook, it will give you like, “Okay, you can run this model with this uh setup type of thing. So you don’t have to think, “Okay, am I am I going to be able to run this or not?” And if you don’t have these computers, you can still download the pre-built binary with the There’s a curl command on llama.cpp in Linux. If if you go to our website and your

02:40 platform is not one of these, um and then you can do run these models, llama.cpp, llama.server, etc. And [snorts] this workshop I wanted to organize these workshops because like I want people to serve their models more intentionally and knowing what is going on one level beyond the you know, clicking, “Okay, chat with this model.” Uh because it gives you much more flexibility and it’s better if you know about it. So

03:10 um if you want to actually serve this, if if you want to learn about like one step beyond chat, this workshop is for you. If you want to learn the tricks to make your go models go faster, it is for you. And if you want to get the best use of your existing hardware because most of with your computer, you can definitely run some of these models. You don’t have to buy a GPU or anything with a larger unified memory or whatever to get started with

03:42 local AI. And if you don’t want to learn about these, it’s okay. Llama.cpp already handles everything for you. So the moments that you actually download Gemma 4, it downloads with its speculative decoding drafter so that you can do the speed up. Let me just briefly check. Okay, we are live. >> [laughter] >> Um So, for instance, how does it look like to to serve a model with speculative decoding? So, on the left you can see

04:12 Music Lemur with speculative decoding, on the right without. We are doing a web search and once the web search is done, you will see the speed up that we are going to get. So, this one already started and as you can see like it’s visible that it’s super fast the generation. And the left one will be done very very soon while the right one will be lagging. So, and and this is lossless, meaning you don’t compromise from the quality of the

04:45 generated text, but you have a minor gap like [clears throat] minor thing with the memory and that’s it. And to understand why is it so and why do we need this thing, I would like to show you very briefly how inference actually works. Like what goes beyond it in a few slides. >> [snorts] >> So, inference takes place in two steps. First one is prefill and the other one is decode.

05:15 And during the prefill, you do you have, you know, like when you enter like so many words at once, you pass it to the model. Model does like a forward pass and it calculates the attention, you know, the and caches these keys and values. So, that every single time you want to do generation, you don’t have to recompute these keys and values again and again. And then

05:46 it also predicts the first token. And this part we call it compute bound because like compute is actually what determines the main complexity, the speed. And then there is decode. And in decode you generate every single token. And to generate these tokens you need to read the weights and then KV cache. And then you up you also write the new KV for the

06:16 new token. And there is this constant travel of the weights basically. So in here we do the prefill, okay, of for the first five tokens and then generate Paris, the token Paris. And then when we do the generation for the next tokens, I am reading the weights and reading the KV cache every single time for every single token. So there is this thing of like reading the way the the weights and the KV cache from the GPU

06:48 memory to GPU compute. And this causes a lot of lag. And this whole travel is what makes the complexity go up. So we want to save up from this. And naturally there is there is some technological advancements to do that. By the way, if you don’t like you don’t have to deal with any of these again because like when you for instance when you go to llama app on like Windows or

07:19 Mac OS when you click a model, you see the KV like the models required memory as well as the amount of KV cache that needs to be allocated for you to have like a conversation of a certain length. Let’s say like 128,000 tokens. So you can see if the if this can actually run on your computer or not. So, it’s all handled under the hood for you, which to me is pretty cool. And

07:50 um And um the the the performance we measure it like with few things. Once we assume that your memory works very well. Um you have like few um durations, let’s say. So, first one is time to first token. So, it is very self-explanatory. It’s the time that that takes until you see the first token. Uh and it is basically

08:21 prefill time uh and then the generation of the first token. And then there is time per output token, which is the average time between the each generated token. And then there is throughput. It is number of tokens per second and I think it’s the one of the most important metrics. That’s why you see it everywhere. And um it just determines the uh user experience a lot. And then there is the end-to-end latency, which is the duration between the time you enter

08:53 prompt until the generation ends. And you can see it here um how the throughput and the time per output token uh works. And in here, like when I talk about the speed up, I talk about the throughput because like throughput is just very, very visible. Uh you have the number of tokens per second that [clears throat] you can you are generating, basically. And how to make it faster? There is like few ways, and speculative decoding is

09:23 the biggest one. Um there is also quantization, but that is lossy compression, technically. Um with the speculative decoding, what we do is that we don’t as as you have seen during the generation during the decode, we do like passes of like big weights between back and forth like in the as well as the KV cache with the whole between the GPU compute and the memory. So instead if you do this with a smaller

09:55 model or like smaller part of the big model, let’s say. If you do this then we are transferring less data and then we can just speed things up and speculative decoding just comes from this idea. And typically what is happening is that like in the classical speculative decoding and there is like so many different techniques LLM is like paired with a smaller model and then this smaller model generates

10:26 multiple tokens. So in here you have like the prompt the cats. And then this the small the small model just proposes sat on my lap and then we pass this to the big model to check and then the big model says okay, I accept this token. I accept this token. I reject this token and then once it makes a rejection because the generation is sequential, you also discard the next token.

10:56 So whatever you do like if I just reject this one the rest of them is discarded. And then it generates one token that is the for the correction and then we repeat this thing. And [snorts] as you can see here for instance like if I generate like four words four tokens and I pass it to the large target model what happened here is that I have generated two more tokens at the cost of

11:29 one token actually three more tokens because this is also the correction. Um three more tokens at the cost of like one pass from the large model and four passes from the small model, which is like this small model is often times is super super small compared to the large model. So, I just save up a lot on the generation and that’s where the speed up actually comes from. And how this correction takes place is that so for instance in the classical

12:00 paper, you have this NV call this small models drafter. Uh so, we say this is like the draft distribution. So, let’s say I have the word cat sat or like I have the cat and then the small model generates sat, ran, slept, meowed for the next word. But then there is like the probability distribution for this and small model really likes sat.

12:30 But then the LLM for instance, it doesn’t like sat, it likes slept, let’s say. Uh what we do is that we take the difference between the distribution of these like generation and then we take like um we we clamp it like with zero and take the maximum and then we normalize it. And this way what happens is that for instance the small the larger model didn’t didn’t

13:01 like sat. It liked slept and then it didn’t like sat. So, sat went to zero, whereas it liked slept a lot, but the smaller model didn’t like slept. The slept got a huge boost and then the corrected token comes from like this distribution. Uh so that the the large models generations are more um favored, let’s say, whereas the small model generations are quite um, penalized.

13:34 And the trade-off here is that we save up on the memory bandwidth, and we get multiples of speed up during the decode. Um, if you just make larger models generate less things, then you read the weights less, the KV cache less, etc., etc. And then, um, there is this thing of like you we technically use more memory because there is uh, this drafter can be either a separate model with its own KV cache,

14:04 or it could be a separate model that can make use of the KV cache of the big model. Um, so eventually you do take like an additional memory, but this is very much negligible. Like if you can actually serve a big model, then oftentimes you can afford this drafter model. And sometimes even like drafter models like MTP heads of like GLM model are within the are a small part in the large model

14:35 itself, so it’s not even that. And like let me give you a few tips. So, the small model has to be good. Uh, so there’s no free lunch. Um, otherwise it will like keep being rejected the generations by smaller models. So, imagine that you you have like the big model generating like one token, and then the smaller model generated like four tokens, and then out of these four tokens all of them gets rejected, then you have you are left

15:08 with like four additional tokens generated from the smaller model, in which it just doesn’t make any sense for you to uh, have this like you are not having a speed up, you have a slow down because you have a small model that takes up everything for nothing because you just keep generating the corrections and these corrections become your generation. Um most of the time model creators like Muse glimmer or Gemma four, uh they do

15:39 release their own drafters. So, at the end of the day you can just use whatever they release um or like you can experiment with other people’s things. Uh but I suggest using them and also your small model has to be in the same domain as the big model otherwise their distributions are dramatically different. Like if your big model is trained for legal and then your small model is trained to write code then they are not a good match or vice versa.

16:10 Um and I suggest generating less tokens with the draft model because then like imagine you predict like eight tokens but like the third is from third it’s rejected the rest is like discarded. And chances of like because there is like a chance of your drafted tokens surviving it kind of compounds in a way towards the eight token. So, you will it will be very wasteful and today maybe um our guests can talk a bit about their experiments with these type of things.

16:44 I want to show you um the techniques but unfortunately I do not have time. I have 3 minutes left. I want to show you how to do speculative decoding with llama.cpp, llama app and transformers instead and give you few um few runs of my favorite models. So, um all speculative decoding techniques like they have names and some of them you will see with

17:15 speculators or Gemma four today. There is multi-token prediction called MTP, there is DFlash, there is Eagle Tree, and there is more, but all of them are like kind of similar, but also different. Um, you don’t have to know about them as much. Um, just I would probably like if you don’t want to learn more about it, just trust the model of nurse own reporting of the speed ups that you can get. Uh, just know that when you go to a

17:47 hugging face model repository on the right-hand side, you have um, like if you go to GGUS repository specifically, on the right-hand side you have the DFlash, uh, which is a technique, um, that you have the MTP, uh, and stuff. And basically, if the DFlash and MTP drafters inside the same repository as this model, then you can see, you know, for instance, how much memory additional

18:20 memory it takes to actually serve them. So, for instance, for this model, which is Qwen 3.5, you have the 8-bit quant quantized version at 28.6 GB memory requirement, but it’s um, it’s uh, MTP is like only 3 GB, so you can definitely afford to run with an MTP, and knowing that the Llama app automatically downloads it for you. Like if you download Gemma for with Llama app, it will automatically download it

18:52 and will not you will not know, which is very cool in my opinion. And, um, if you want to serve them, uh, so in Llama serve, uh, you have if if the model is in the same repository as the if the drafter is same repository as the main model, which we call the target model, You can just give the a model ID and then you give the technique like I’m okay, I want to serve MTP drafter and then number of tokens to be generated

19:23 with the Lava serve. If they are in different repositories, you can provide the other repository that’s the drafter is at with the HFD um argument. And lastly, you can also do this with Transformers. Uh with Transformers, the pipeline abstraction handles it. So, for the classical uh speculative decoding, you have the model and then you specify this drafter model. We also call it the assistant. There is a naming issue.

19:54 Assistant model like for instance, you can combine Lava with a smaller Lava uh for drafting. And then you can initialize the pipeline like that and then infer as usual. And for multi-token prediction heads or layers, sorry, um this in this technique, which is different than the Gemma’s MTP by the way, it’s the layer MTP layer is part of the big model. So, you load only those weights and then that MTP layer will be

20:27 used. Um but you need to specify use MTP equals true so that those MTP layers are used. And this applies to I think Gemma 3.5 and GLM A 4.5 A A A I think. Um yeah, like you can just do this. I’m just load the normal model like the GLM 4.5 A with no assistant model. And these are my model model recommendations for you to run with the models and without further ado, I would

20:59 like to have um Eldar in my um live stream. Hello again. Let’s me put your um presentation. Everything’s fine. I’m going. >> [laughter] >> Hello and see you. >> See. You can see my slides, right? >> Yes, yes. Everything’s fine. Thank you so much for joining us. >> So, thanks a lot for the introduction. So, yeah, my name is Eldar Kurtic. Um I

21:31 work as inference research lead at the VLM team at Radboud and the Institute of Science and Technology of Austria. And the topic for today um is like introduction tutorial to Speculators, uh which is a library uh inside our VLM project. So, for I I assume at this point everyone heard about VLM. Uh it’s like one of the largest open-source projects um which is uh with with the main purpose of making um

22:02 inference with LLMs very efficient and making best use of your of your hardware. Uh one just kind of thing one one most important thing to mention here that not everyone might be aware of is that VLM is not just inference engine. It’s the entire ecosystem around the inference engine. So, in addition to VLM, uh we have like um LLM Compressor, which is library for quantization, for pruning, knowledge distillation. We have Speculators, which is the topic for today where we are enabling training of the Speculative Decoding models. We have

22:32 Guide LLM for benchmarking and so on and so forth. So, it’s kind of like the full full ecosystem around this. Uh the other important note also to mention at this point is that uh most of the things that we will see today that I will present today um is based on the results from VLM, uh but all of these findings should be applicable to any other inference engine or any other deployment environment that that you care about. Um so, uh under the VLM project uh {slash} Speculators, you’ll find a library.

23:04 The main purpose, as I mentioned, is training speculative decoding models. All of the models are in a Hugging Face compatible format. This means that wherever you can use Hugging Face models, you should be able to use speculators. One for example use case is in the llama.cpp project. We have seen people just exporting the speculators models into the GGF format and then being able to use them to use them there as well. Initially, models are exported into the format suitable for VLM.

23:35 But, as I mentioned, we are also like we are working on a standardization between many different inference engines at this point, so we all have the same the same format. So, you don’t even need to do these conversion steps at at some point. The main feature of speculators library is that it implements a lot of these state-of-the-art algorithms for speculative decoding, such as Eagle Tree, MTP, D-Flash, D-Spark, D-Flash 2. So,

24:06 the main kind of difference between them is that Eagle Tree is like a standard auto-regressive speculator. So, we are speculating or like predicting one token at a time. Whereas, D-Flash, D-Spark, and D-Flash 2 are predicting multiple tokens at the same time. So, they would usually predict either a block of eight or 16 tokens. And I will not go into MTP, so the next the next speaker will will cover this part. What we’re going to focus on today are D-Spark and the results with D-Spark and

24:37 the D-Flash 2 because these are state-of-the-art techniques at this point in time. And both of them are built or like are kind of extensions of the D-Flash. So, D-Flash is a approach where we train the model to predict hidden states of the the follow-up tokens in the future by taking the hidden states of of our target model. So, what what we would do in the normal pipeline, we would extract hidden states from our target model, we would bundle them together somehow, usually concatenate,

25:08 maybe normalize, and then project. And these would be the inputs into like a speculator model, in this case I’d say D-Flash, and at the output of this we would get hidden states for the next for the next set of tokens. Um DeepSpark and D-Flash 2 introduced additional one kind of like improvement over D-Flash related to the dependence of the predicted token. So, in D-Flash because we are predicting in parallel multiple tokens at the same time, the main problem is that once we

25:38 predict eight tokens in in one forward pass, there is no dependence between them. So, when we are predicting in in that way, token two is not aware of what what has been predicted at the token position number one, which is of course is bad for text generation because usually we would like to have this kind of causal dependence. Um when we’re predicting token N based on all of the previous tokens in the sequence. And DeepSpark and D-Flash 2 introduced this additional kind of novelty where we on top of the more or

26:10 less standard D-Flash backbone architecture, we have one small loop or one small architecture which introduces this causal causal dependence between the tokens, and this allows us kind of to speculate more tokens at the same at the same in the same pass. So now, we’re going to take a look at how how one life cycle of a speculative decoding model looks like. You should not try to like read everything in this this diagram. I will try to break it down like a

26:41 in parts. The first part of the pipeline is regarding the data preparation. >> [clears throat] >> So, one one more important thing regarding the presentation, I will not be showing any code snippets throughout the presentation because uh we have very good docs uh and tutorials in the Spector’s library. So, you can just go there and see how the code snippets for data generation would look like, for data preparation, uh for training, and stuff like this. I’ll try to give for each of these stages one or two uh tips or tricks, so to say, which

27:12 are not uh documented anywhere in papers, which are just kind of somethings we learned learned along the way either for training or for deployment of of Spector’s. So, the first stage of the pipeline is prepper data preparation part. So, because in the speculative decoding uh we are training these Spector’s to mimic behavior of their corresponding target model, uh we need to generate the data with our target model. So, usually how we do this, we go to Hugging Face Hub, we pick a data set that is a representative data set for the use case

27:44 we care about. Usually, let’s say we can take maybe perfect blend data set, and we discard all of the answers in this data set. Um we get we we keep only the prompts, uh and then we regenerate all of these answers by prompting our target model. So, we take all of the prompts from all perfect blend, we send them to a let’s say a VLM server, which is serving our model here, and then we collect all of these all of these responses. Uh once we do this, uh we do the next

28:16 stage, which is like a data preparation phase, where we can uh specify whether we want to train uh either on prompts or on the outputs or like the assistant answers. Usually, we want to mask uh the prompts. We don’t want our Spector to train uh on this on this data. We just want to focus on the assistant uh assistant answers. Uh then we compute also uh some somethings like we can impose some constraints on how many turns we want to have in each sample. Um we We also do

28:46 uh compute some statistics about the tokens which appear in the data set, um which is for example very relevant statistic if we are deploying extremely small speculator models. Um if they’re very, very small, then uh their LM head, which they usually share with the original target model, might become a bottleneck for inference, and therefore we wanted to prune the LM head uh because maybe in our data set we will never see some specific set of tokens, and therefore the speculator model doesn’t have even try to predict it uh

29:16 predict those. Uh so, that that would be the data set preparation stage, and in this stage the main the main kind of thing to be aware of is that this is very compute-heavy kind of part of the stage uh of the of this entire pipeline because we have to send, for example, perfect blend has like like a little bit more than a million samples. So, we had to send we have to send a million um prompt million around million prompts to our model, and then to wait the model to complete each of those. And this becomes very, very, very problematic because, for example, for

29:47 Qwen models, uh for for for which we know that the uh they like to think a lot, uh they generate a lot of tokens, this becomes very, very intensive operation, specifically when we have like some some some samples that require a lot of thinking. Um in order kind of to get around this constraint, uh the first kind of tip or trick um is to like try to use data distillation. Uh here what we mean by data distillation in the context of speculators is uh the setup where we

30:18 usually uh get models in a batch. So, for example, like Qwen models Qwen model releases are usually like this. We get a full family of models from very large MoEs to very small uh dense like a decoder-only models. So, usually in a normal speculative decoding pipeline for each of these uh models in a family, we would regenerate the full data set and then train on that data set because I I that would be ideal use case uh so that we capture the uh text distribution from each of these models that we want to train a

30:49 speculator for. However, uh this of course is is very costly. It takes a lot of GPU resources to do for each model. So, one thing we wanted to try, we wanted to see if we can generate the data with the largest model in the family and then train speculators even for smaller models on the data uh uh uh produced by the largest model. Uh and our main goal in this experiment was just to get results which are close enough so that we can live with this approximation. Uh but uh like as it turns out, it seems like this actually

31:19 even improves uh results uh uh for speculative decoding models. One such example in in the in the in the in the picture here is Qwen-3-235B model. Uh so, we generate the data with this model and we also generate the data with Qwen-3-30B model. And then we train uh a speculator for 30B model um uh uh by following the data set from 235 billion model and by following its own its own generated data set. And then we get something like this. On Y axis uh we

31:51 report the mean acceptance length. So, you can look at this as kind of like accuracy, average accuracy of a speculator on a couple of different benchmarks. And then these benchmarks are kind of standard uh benchmarks coming from the SpecBench SpecBench paper. Uh and we see that uh the blue bars are are slightly higher than the the orange bars. And blue bars represent this kind of cross distillation where we train a speculator for smaller model uh on a data from a larger model. And we see that uh the the accuracy results uh improve very very consistently. And it’s

32:23 it’s been very unexpected uh finding because usually we would expect that the data when we are training a speculator, the best use case, the best data for training uh of that model comes from that from that model uh itself. Uh but here we see that uh they’re taking a data from larger models seems seems to perform slightly slightly better. Uh usually I think we hypothesize at this point that this is mostly because um all these models are a kind of distilled versions of each other uh or like uh distilled versions of the of the largest largest one in the family.

32:53 And then we’ve been also able to replicate more or less the same findings on GPT-NeoX, so we took GPT-NeoX 120B uh and 20B and then we got uh either the same uh or slightly better results uh by just uh regenerating the data once uh for the largest model and then training the smaller model in family with it. The next stage in the pipeline uh is extraction of the hidden states or at this kind of uh point we have a data set. Now we have to figure out how we want how how we’re going to train the

33:24 model. There are two choices uh we have here. It’s online and offline training. Uh >> [snorts] >> offline training, so in order to train a speculative decoding model like a Deep Flash or Deep Spark, uh we have to extract the hidden states from the model. As I mentioned in the beginning, these speculative models are trained by extracting the hidden states from the larger model and then uh kind of combining them in some way and then predicting the hidden states for the follow-up follow-up tokens. Uh so what we need to do here for each training sample that we’re going to train on, we have to pass it through the target model

33:56 and then we have to collect the corresponding hidden states from a couple of layers and then we have to pass them as inputs to a speculative model. In the offline setup, um what we would do, we would take all of our training samples, we would pass them all through the target model, we would collect all of the hidden states uh for this model, we would save them on disk, and then we would have this like a fixed set of inputs saved on disk, and then we would train our model on this. And this is great for very quick runs uh because our our like input data set is

34:26 already generated and it’s stored on the disk. However, the main problem with this is that uh this consumes a lot of disk storage. For example, for quant 380p model on a perfect plant data set, in order to store the hidden states for just five layers to train a speculator model, we would need like tens of terabytes of the data data data storage. And this is extremely huge kind of requirement for for any any infrastructure and it’s usually very hard to move it around and and work with that.

34:56 >> [snorts] >> So then we kind of like we have this complementary kind of approach which is called online training approach where we uh serve a model. So we instead of training on all of the GPUs that we have, we dedicate a small pool of GPUs to serve the model and to stream hidden states to our training script. So we would allocate some GPUs for training and during the training phase we would send the training samples to our BLM server which extracts the hidden states for those training samples and gives them back to

35:27 our trainer which then trains trains the speculator model in this. And this is great because now we don’t have to stay save anything on disk. So we don’t have this extreme like requirement for very large disk storage, but then comes with the with the drawback that we now have to allocate some GPUs for data generation and we cannot use all of them for for training. >> [snorts] >> So depending on the size of the model that we are training, depending on our infrastructure, we would pick one one of these two. Offline because offline

35:58 offline training is very attractive in the sense that it’s much going to be much faster because we will not be bottlenecked by the online data generation part and we’ll be able to iterate very fast, but unfortunately bottleneck with the huge disk requirements makes this a little bit impractical. So one thing we tried is to quantize the hidden states to reduce the storage requirements for this. So here we are looking at a average acceptance rate or like average accuracy across the spec bench,

36:28 many different tasks in the spec bench on quant 3 HP model, D flash speculator, and we just trained it for half a million samples. And then we quantized in each of these runs, we quantized the hidden states down to eight bits, four bits, three bits, and two bits. And if we look at the average plot in the bottom right corner here, we see that we we can push quantized hidden states down to four bits and still get more or less the lossless the lossless acceptance length or lossless accuracy.

37:00 However, we have to use some tricks from the quantization world such as Hadamard rotations in small groups of of weights in our in small groups of hidden hidden states in order to smoothen out the activations. But kind of given given those tricks, we we we can reduce the disk storage requirements by more or less four four x. The next stage in the pipeline is then training. Once we decide to train our model, we have to pick the the speculator that

37:30 we want to train. Usually these days we would decide between these park or D flash 2 with a like a like a small tendency towards D flash 2 given that we can can can tune it properly because it usually outperforms these park. And kind of during this this stage here, we also have to make a couple of choices because this is now the standard like a like a LLM training except that now the inputs are hidden states and we are predicting also hidden states at the at the output side.

38:01 There are some standard choices that we have to make as in the normal LLM training such as learning rate, learning rate scheduler, batch size, sequence length, like the standard the standard kind of LLM fine tuning hyper parameters. One thing that we noticed in this kind of like a I now kind of going to the tips and tricks for training part. Uh one thing that we noticed is that most speculators these days uh are being trained with Adam W. Uh and from the LLM pre-training people,

38:33 we know that Muon is a much better choice um for speculative decoding training. Uh it of course it does come with this like a slightly higher cost per optimization step, uh but it has much better uh convergence. Uh like it’s much more uh it’s much more uh data efficient uh optimizer. And therefore, if we apply it for speculative decoding training, we do see some very nice uh and easy wins. So, here we are looking at Qwen-3-8B and Laguna-XF2 model, and we are doing

39:04 ablation studies of sweeping over different learning rates and then measuring the validation validation loss. And we see that even the worst uh like uh Muon checkpoint is better than the best than the all either the same or slightly slightly better than the best best uh Adam W checkpoint. And it’s very easy to use, specifically for um uh speculator training where uh these models are uh rather small, uh and this like a small kind of impact on the optimization step uh from Muon is not not going to impact

39:34 our runtime uh significantly. Uh and then if we now take these validation metrics or like these models and like go like from validation metrics to actually like some out-of-domain benchmarking, uh and here we are looking at the SpecBench, uh and then we plot uh SpecBench versus the validation um accuracy. We see that like very nice and clean separation between all of the points uh produced by Adam and all of the points produced by by Muon, and very nice kind of tendency to to this like uh upper

40:04 right corner where we actually want to be, where we have a higher validation accuracy and also higher test test accuracy. Uh the next kind of trick for for training or uh is like uh the choice of the attention because when we are designing our speculator model, it’s a standard more or less very very standard like LLM like architecture. We initially started with the llama models, but then community converged on when when layers. Uh and one thing we inherited from from

40:35 from from the when is is this like a full full attention mechanism. These are like the older architectures when when when actually used full attention. Um so then once we train a speculator model, we usually train it with uh sequences which are not that that long. Usually uh speculative decoding uh pipeline doesn’t get as many resources as the standard LLM pre-training. So we are not able to like train at 1 million context length and so on. So we usually train at 8K, 16K, 32K depending on um

41:09 what our hardware requirements allow. Uh and then if we train with a full attention uh full attention mechanism, uh we usually see like an huge degradation of the accuracy of the model as we start increasing the context length that we are testing the model on. >> [snorts] >> Uh and this is then of course the problem uh specifically for these like long reasoning and coding coding coding setups. So if we swap the full attention with

41:39 the sliding window attention, uh we get and this model specifically here in the graph has been trained with sequences of 4K uh 4K length, uh we see a very stable we see some degradation of course, but then we see uh very stable performance at longer sequence lengths. And one additional great feature that we get out of this is that uh sliding window attention has this like a bounded constraint on the KV cache uh and therefore it gives it it gives a little bit more space uh to the KV cache of our target model uh

42:10 compared to the to the full attention part. Uh, and then the last stage is of course serving the model um, and validating this. So, uh, in the Speculative library there are like all there is a lot of like um, examples on how to benchmark the model with Guy TLM um, on many different benchmarks like Spec bench, Speed bench, MRCR for long context and so on and so forth. Um, and if you want to deploy the model it’s as simple as just running VL app serve and then providing a path to our

42:41 checkpoint. One important thing or kind of the trick uh, for for serving the model uh, is like the the like once we decide to deploy a Spec decoding model we have one hyper parameter to tune and that hyper parameter is uh, the block size. So, how many tokens we want to speculate with this model. Usually we train like a deep spark, deep flash too to predict either eight or 16 tokens. Uh, and when when we deploy them we usually want to like just use the same number like during the inference time.

43:12 But one of the major problems um, uh, uh, uh, uh, as we have seen in the in the previous talk like uh, usually once we increase the number of tokens so once we go to the higher steps like step uh, prediction step five, prediction step six, seven and so on um, the acceptance probability that we’re going to accept these tokens like goes goes very quickly to zero. Um, and this is especially uh, uh, especially noticeable once we deploy our speculative decoding models

43:43 um, in a setup where we get prompts which are coming a bit out of the which are out of distribution from what the model saw during training. Um, because during training you the these models are extremely small so we cannot expect them to generalize on unseen data sets or unseen domain in unseen domains. Uh, and this means that as we start like like like dropping or like rejecting more and more tokens. That’s going to be a wasted compute for verification because our speculator will speculate eight tokens. We will take all

44:15 these eight tokens and pass them to the target model for verification. So, we’ll do a forward pass with eight tokens through a through our large like a verifying model. And then, as we increase the batch size, this becomes even more larger problem because we now have like many more concurrent requests coming in and we are doing a forward pass for all of them at the same time. Even though we we we know that some of these tokens in the block have very low acceptance probability. And for this use case in real time, there is something called adaptive verification

44:45 where we can use models models certainty about the the tokens that it’s predicting and use that information in combination with the information on the run time. How How long does it take to run forward pass with one speculated token, with five, eight, 16? And then, combine these two pieces of information and actually decide before we actually do the verification, decide for each request how many tokens we’re going to verify. So, then we’re going to end up

45:16 with some some requests where we validate only four tokens. For some requests, we would validate eight tokens. For some, we would validate 15 and so on and so forth. And then, if we plot this distribution represented by the blue blue plot here with adaptive verification versus any of the fixed choices we could have for this speculator here that supports up to seven speculated coding tokens, we see a nice like a Pareto frontier curve on this throughput versus interactivity graph

45:46 where the best kind of position is to go to the upper right corner where we have the highest throughput and the highest interactivity. And now, to put all these these results together, just kind of to give you some like like brief overview of what you should expect to see in practice. So, for example, here we are looking at throughput versus interactivity graph for Gemma 3 so, as we go, we start we evaluate this model by sending first like a single stream request or one request at the time and then we go from right to

46:18 left and we increase the number of requests we are sending to the server up to the maximum load that the server can take. And then we observe that as we increase the number of concurrent requests, interactivity or like number of tokens per second per user decreases because we have more requests so the server is more busy with more requests, but the overall throughput goes up. And then we train a Deep Spark checkpoint or Deep Spark Speculator for this model and then we replot or rebenchmark the same the same

46:49 the same set in the same setup and then we see like a huge throughput and interactivity gains for single stream performance like close to 4x and as we increase the number of concurrent requests, this goes down to more or less 2x. Then we can also take a look at one small model in this case Gemma 4 31B model also in VLAM. If we do the benchmarking without Speculator, we would get a graph that looks like this. Then this model comes with its own MPP layer

47:22 which you will hear more about in the next talk. We would get like very nice speed ups in both single stream and the higher high high concurrency regime. Um like going from 3.5x to 1.6x. And then if we train a Deep Flash Speculator, we usually would expect to get even higher speed ups at the low concurrency settings where these parallel drafting techniques work great, but then as we increase the number of concurrent requests, MPP actually outperforms this.

47:52 And the last important note here like the these parallel drafters are also not the golden bullet. We should not expect them to perform great in all scenarios. This here is a very nice use case because I’m evaluating on math reasoning, which is relatively low entropy and well represented like a domain in our training data set. But if we for example evaluate the same model, the same speculator speculator on high entropy set up in a high entropy setup where the model hasn’t seen also like

48:23 what it’s like slightly out of the domain, data to what the model was trained on, we would see something like this. MTP still has like very consistent gains, but then the D flash two speculators would be more or less either the same or the worse choice than MTP. And this is mostly because these are very very small models and we cannot expect them to generalize well compared for example to MTP heads which are trained alongside the original model and they have seen much more data.

48:54 I’m going to skip this one. So domain because don’t think we have more time. But the main point here is that we can combine quantization and speculative coding so we can get compounded benefits from from both techniques at the same time in VLN. And if you want to start, Red Hat AI Hugging Face Hub offers a very like easy entry point because our team is actively releasing a lot of these speculator models for the popular open weight models. So you can find models on a Hugging Face Hub.

49:26 Usually all of these models have very detailed model cards and that’s why also I didn’t want to go into the like the code snippets because every single model card has exact code snippets which were used to produce these models so you can just copy paste them and use them in your in your own setup or like tweak them for your own use case. However, as I kind of showed before, if you have some specific data data you care about, some internal domain specific deployments, you should not using these models and you should be training your

49:57 own speculator on your own data set because that’s that’s going to be the best setup for for speedups. And yeah, that would be all from from my side. >> Thanks a lot. I I think we can have like one question and then maybe you can answer questions in comments. I think there is like two or two more. So, and I’m also curious about this because I never experimented with greedy decoding. Is output guaranteed to be I token-for-token identical to target

50:28 model alone or can batching kernels change it? >> Yeah. So, this is this is a very good question. So, speculative decoding is a lossless in a way that the output distribution the output text will be coming from the same distribution. But coming from the same distribution doesn’t mean it’s going to be token-by-token the same. Usually for greedy decoding, you would we would expect to get the exactly the same outputs, but due to differences in kernels, this might not always be the case. So, I would like for greedy

50:59 decoding, you can I mean we even have some examples in speculators where you you’ll actually like you can run them side-by-side and kind of see where they start diverging. Usually what you will see is like on shorter sequences, they’re going to be token-by-token the same, but for longer ones because of the CUDA graphs compilation and kernels, this this will change with with longer sequences. So, they will they will start deviating at some point. >> I have like one more question that I’m super interested about, so I’m going to ask. Is it possible, which is

51:31 interesting, I never thought about it. Is it possible to train a speculative decoding like model native heads for speeding up prefill? It’s interesting. >> [laughter] >> That would be interesting to to to see. I I I’ve never thought about it. >> Yeah, me neither. Thank you so much. Then maybe we can have Martin. Thanks a lot, Eldar. Uh I have put Speculative Project on the on the stream and then maybe we can send

52:03 the slides to the people. >> Yeah, yeah, of course. Yeah. Thanks a lot for the invitation. I’ll take the other other questions offline. >> Thanks a lot. So, let’s have Martin. Hello. >> Hello. Thank you for having me. >> Thank you for coming. I’m a huge fan. Um also for Eldar as well, but um can we have your slides? I or screen. >> Let me >> I’m being so starstruck from this uh live stream today.

52:33 >> There’s so many incredible people here. I mean >> I know. >> And I have to follow them up. What’s happening here? >> [laughter] [snorts] >> Just added. I will leave you with your presentation. Many thanks. >> Thank you. Uh it’s it’s wonderful to be here. It’s also amazing to follow up some some amazing people because I learn a lot from uh seeing them talk about how they approach it and feel them for instance that we just discussed is an is an amazing project. Truly incredible to see

53:03 what’s happening over there. Um and fortunately a nice segue into well, the Gemma 4 drafters which I’m I’m obviously going to be talking about. And what I want to do is I want to talk a little bit about how it works under the hood. For those who know my my work, that that’s what I like to do, right? A lot of visuals showing you how the architecture of these things work uh and what that means. And multi-token prediction and we’ve already had a lunch of a lot of information about that, but uh to me that’s a a rather special

53:35 thing. It it it’s not perfect, but it’s almost a free lunch of a speed up, right? On the left you have no MTP for Gemma 4 and on the right you have it. And now as mentioned before, with short sequences it tend to be token by token identical. And it just speeds things up. Right, we get almost the same output or specially in most cases the exact same output but it just makes it go faster. Now I’m saying not a free lunch entirely because as we

54:05 seen before there are very specific circumstances under which it might not be the perfect solution. Technically it increases, you know, the memory that you’re using and especially in memory constrained environments like on mobile it’s not easy to, you know, jam an MTP in there even though it’s fairly small. But in general this is something I I get really excited about to go in depth into what this architecture means and how it works. What I’ll be doing is I will compare a

54:36 Gemma 4 model, the smallest one E2B, which is a 2 billion parameter model or effectively 2 billion parameters, with its draft model that was specifically trained for this E2B model. And this draft model is less than 100 million parameters. And from the top of my head I think it’s like 77 77 million parameters. So very very tiny for what is already a very small model. And I want to explore how that works, right? What’s the architecture of Gemma 4 and how does the draft model use that,

55:08 leverage that to speed up inference. And to do that, obviously, we have to start with the architecture of Gemma 4 because that’s what it uses. And as most models it start, well, with the token embedding layer. Produces embeddings for the tokens and passes that through a number of decoders, in this case 35 of them. And as with decoders that information is then being processed through attention. Now with Gemma 4 a decode layer might either have local

55:40 attention or global attention. Every four layers, you will have local attention. So, that was the sliding window attention that we saw before. So, four local attention and then one’s global attention or full attention. Again, four local, one global. Four four local, one global, etc., etc. And that becomes important later on because the draft model has to do something with with that way of interleaving different types of attention. Now, as always, this information is then

56:11 being processed by the forward neural network with some normalizations here and there. And well, specific to the E2B models, the E2B and E4B models, we have per layer embeddings. So, those are the token embeddings that are very specific for each single layer. So, the token embedding layer normally has one embedding per token, but the per layer embedding has one embedding per token per layer. So, that provides it with more dimensional information that we can easily add without the need

56:43 to put it into VRAM or RAM. Can be essentially queried from your flash storage, for instance. All right. After that, we have activations, of course, right? We have the hidden states, final hidden state before it actually produces the token. This is the architecture of Gemma 4. Now, as I mentioned before, the reason why I’m showing this is because the draft model that was trained specifically in conjunction with Gemma 4 uses a very similar architecture because they they need to connect together in

57:15 order to make them as efficient as possible in what are very constrained environments. A 2 billion parameter model is kind of meant to be used on a phone, for instance. So, that has limited RAM to use. The draft model, therefore, needs to be as small as we can make it, humanly possible. And to do that, well, you kind of mirror the architecture that that Gemma 4 model is using. How does that work? Well, we start again from the token embedding layer. Now, it shifted a little bit to the

57:45 right, and that’s because it it’s not only the token embeddings that it uses. It also used the activations from the target model. And as was mentioned a couple of times before, the target model is the main model, this case the Gemma 4 model that we want to speed up. So, it uses both, and they are quite big. So, if we were to um process them through the model, it wouldn’t be a 100 million parameter model, or you know, what was it? 77? Uh it would be way bigger. So, it needs to be projected down to

58:17 compress that information and make it more efficient, because that’s the entire purpose of a draft model, right? A small model that can predict or suggest very easy to predict tokens. And it helps if it’s small, because that actually does uh creates a speed up. So, only 256 values, and then it passes through four decoders only. It’s a smaller model, but it uses the exact same architecture, right? Local attention or global attention, the same normalization

58:47 uh happening there, feed forward network, etc., etc. So, it mirrors that in order to reproduce kind of the way it’s doing its processing uh to get to a very similar distribution of predictions. And then finally, we have now the same sort of output. And these are then the two architectures that we have. Now, as I mentioned before, there’s this local and global attention that we’re interleaving in a way, and Gemma 4 has a

59:18 bunch of decoder layers, and it starts always with four local attention, and then one global, four one four one four one. So, it ends with a global attention layer. Which is full attention and which is which improves performance. Now, the draft model essentially does the same thing, but it’s a very small model, so it only has three local attention layers and then a global one. Whenever the main model, this target model, Gemma 4, is processing information and predicting tokens, it’s

59:49 generating the the KV cache. And that’s information that the draft model can leverage to make its predictions. So, it attends to it cross-attends to the the the KV cache of the Gemma 4 model, of the very final global attention layer. That’s where most of the processing has already been done, and we want to match global with global, or full with full. And the same applies to the local tension, of course. There are three layers in there, so let’s pick the final local tension layer

1:00:22 as a way to get the most information into the draft model without the draft model doing the computation, in a way, right? That’s what Gemma 4 model did, and through this cross-attention mechanism, um it can leverage that and also remain relatively small. So, we now have these two models, right? So, how does then the um an actual round work? How does the draft model work in conjunction with this Gemma 4 model? To explore that, we’re going to go

1:00:53 through, well, one round of processing. And through that one round, you will see what’s happening as a Gemma 4 model creates its state, and as the MTP model, the drafter, takes it over, and then slowly but surely creates a bunch of candidate tokens that the Gemma 4 model can validate in one go. So, let’s see how that works. We start with the target model. The target model always needs to start, of course, because it has to create its

1:01:23 state and from there on we can fine-tune it. Or at least we can predict some candidate tokens. Let’s assume we start with an input token, token actions. It passes through the entire model and Gemma 4 generates the token speak. Action speak. After that, it has created its activations, its hidden state, right? And that can be leveraged by the draft model because after this we can leverage draft model to come up with a number of

1:01:54 tokens that Gemma 4 can verify in one go. To do so, after generating speak, we can leverage those in an those activations, its hidden states, and give it to the draft model, like we saw before. Combined with token embedding layer of the draft model, these are projected down to 256 values as a way to make it efficient, to

1:02:27 compress the information that’s in there and speed it up, but also as a way to reuse what was already generated. And that’s what you will see in in many techniques also outside of MTP, is we’re trying to reuse as much of the very costly processing that has been done so many times before. And in the case of MTP, why not use what was already there? Now, we have those 256 [clears throat] values for, you know, given token. Um and then it passes through those four

1:02:57 decoder layers. Three of them are local and then one global. It’s a very small, it’s a very efficient process, and then the rest of it is essentially the same. But it generates a draft token or candidate token. It generates a token that we don’t immediately accept, but we have it there to validate at a later stage. So, this is just one token. Now, what if we want to do two, three, four, or five tokens?

1:03:28 Well, in the next stage, the draft model has has changed what the state was after Gemma 4 was generating the the the the token speak. It has generated louder, and it’s not like the Gemma 4 model now does another move that we have to wait until the draft model is done predicting some tokens. So, we use its activation projected up so that it can be combined with the tokens again. And what we now have is we have the

1:03:59 original input action speak that was the state at the time of Gemma 4. And then the draft model, well, first predicts louder, then does the projection up into well, 1,536 values, predicts another word, then, and continues doing on this process until it’s done with now the the the the length that you want it to be. For this example, it’s three, but it’s very common to do four or eight.

1:04:30 Anything more than eight typically will result in many tokens not being accepted. So, often a very safe bet is four, depending on the use case, as we saw in the in the previous amazing talk, by the way. So, these purple tokens here, let’s call them candidate tokens. We We haven’t accepted them yet. The draft model has generated them quite quickly because it’s a very small model, right? Less than 100 million parameters. So, we can kind of assume that it goes much, much faster than a 2-billion parameter

1:05:01 model. And then we have this this new state, action speak louder than pants. And this state, this information, is then being passed to well, the Gemma 4 model. And the great thing about doing something like this is it can validate all of these tokens at once. It can look at the entire sequence and say, “This is what I think of it.” So, it passes through in a a single forward pass. And through that forward pass, it takes those three candidate tokens louder than pens, looks

1:05:33 at louder and says, “Oh, this seems reasonable.” “Then seems reasonable, but pens not so much. It’s kind of weird to put there.” Which is fine because we reject it and we don’t use it at all. It would be nice if we can accept anything. But if that always were to be the case, then why not just use the 100 million uh parameter model? There’s a reason why these bigger models are smarter. So, pens is not accepted. The thing is though, is when you go through this

1:06:04 entire process, a single forward pass kind of means we can still generate another token because Gemma 4 did its its work. It creates activations and let’s put the language modeling hat to work. So, let let it predict another token at the place where pens was rejected. So, then it generates another token, which well, obviously is accepted because the model generated it. So, at the cost of one E2B forward pass, it kind of

1:06:38 generated three tokens. I’m saying kind of because well, it didn’t technically generate louder and then. The only thing it did, it validated them. It looked at those and said, “Well, this is reasonable. Let’s use that.” But there’s always one free token in the way and it’s not free. It’s still forward pass. But because it’s doing forward pass, it can still generate another one. Now, and this is what I I I truly like about MTP because it’s a very elegant

1:07:09 way in general to make sure that this model goes faster. Because principle still is is that the output should remain the same. And indeed when you look at kernel level and and CUDA, that might change slightly. But assuming most of it stays the same. BC near identical output. And that’s the great thing about using something like MTP in general, whether it’s this or Deep Flash or any of the other amazing techniques we’ve seen so

1:07:40 far, is that the only thing you need is room for 100 million parameters in this case or less. And add that to your environment and you will get out generally at least a 1.5 x speed up depending on the use cases. The output remains the same. Um and it’s a very neat way of using models already there because there’s so many tokens that are so incredibly easy to predict.

1:08:11 And that also means that the speed up that you will see tends to be in very structured use cases. So when you use the coding use case or mathematic use case, especially coding for instance, it’s very structured. It It is a specific language that needs to adhere to certain words and tokens that need to be in a very specific order. And that order is that not really the most difficult thing to to predict because it’s always in that order. There

1:08:41 are never any exceptions because otherwise you would get errors. So those specific tokens are so incredibly easy to predict and that’s why MTP works so incredibly well for those use cases. On the other hand though, um when you have creative use cases where you’re creating, you you a story or poem or what have you, these tokens are a little more difficult to predict because we’re aiming for a little bit more randomness in the output. Gives us a little bit of flavor, right? Um

1:09:13 and then you will see uh smaller speedup. But it’s still a speedup in most cases. Uh I haven’t seen one where every single draft token is uh is ignored, is is rejected. Now, so we have this E2B model model, right? And I purposefully chosen this one to showcase because it’s in very constrained environments. And these constrained environments needs to be as efficient as possible. So, for the smaller model, smaller

1:09:45 models, the 2 billion and 4 billion parameter models, we also have something called the efficient embedder. And that’s yet another trick that we try to do to make sure that the drafter is as quick as possible, uses as little RAM as as as we can. And that becomes an interesting challenge because you generally don’t have that at larger sizes. It’s larger sizes, you going to assume that people have just more compute available, and you don’t have to really look for one or two differences in the

1:10:17 in gigabytes of VRAM, for instance. But even 100 million parameters or uh a little bit slower makes such a big difference on these smaller devices. So, one of these techniques that we implemented was the efficient embedder. Now, what does that mean and why is it called efficient? Well, let’s assume that we have our draft model here again. Actions speak is the input and louder is the output. And at the stage of the language modeling head, it’s doing something that we all know

1:10:47 uh well. We take the hidden state and we multiply it with uh well, the token embeddings. And through that, it generates a logits for each single token the about confidence in a way to say that well yoga is a kind of a weird token to use so let’s use them for instance a little bit simplified but you know this works. The language modeling head is essentially un-embedding the hidden state that has generated into logits that we can actually use to say okay but

1:11:18 this token well that seems like a nice one to predict so let’s use that. The thing is though we’re trying to optimize every single thing in there at least as far as that’s possible to do. And it means that we have to look at this matrix multiplication because it’s still expensive. 256 dimension is smaller significantly smaller but it’s not nothing. Right? If you multiply them together that’s still

1:11:49 a lot of processing that needs to be done. And the entire idea of the draft model is need to predict stuff as quickly as possible. At good quality of course. In order to have it validate by the larger model so can we do something with this matrix multiplication? And as it so happens yes especially for the smaller models. What we can do and that’s a very interesting trick at least I think so because because I like clustering is we cluster.

1:12:19 As we cluster all of these tokens together into groups of semantically similar tokens. When we do that we look at them and we generate an embedding for each single cluster and we assume that that embedding is the most representative embedding for each of those clusters. There are many different ways that you can you can do this and you can do it for example by simply taking the centroid or more sophisticated techniques but you create a representative embedding.

1:12:52 You extract them and you essentially create a cluster lookup table. A table where you can see the embedding for every single cluster. And because they’re clusters, there way fewer embeddings in there than we would have with the token one that we saw before. Instead of 200 K tokens, it’s now a couple of thousand at most. So, why do we do this? Why would we create these clusters in the first place? Well, looking back at the

1:13:23 language modeling at and the multiplication that it has done, what we can do now is we can look at those 262,000 tokens that matrix and we replace it with cluster embeddings. When we do that now and we multiply this hidden state with these cluster embeddings, we don’t get token logits, we get cluster logits that essentially say the following. The likelihood that the next token that we should predict

1:13:54 is in one of those clusters is this and this value. In this example for instance, we say, “Okay, the the the the token is most likely in cluster three.” But what we can do is we can choose for example two cluster or three or four. But you take a couple of them because well, sometimes it might not be in the the most probable cluster. It might be the second or third. But by doing that, you’re essentially cutting out like 90% of the of the tokens and only focus on well, the most

1:14:26 probable ones. That means we can now extract the token embeddings for for instance those top two clusters. And instead of having those 262,000 um tokens in there in that matrix multiplications, we now have only those for two clusters. A couple of hundred or a couple of thousand tokens at most. And in turn, that will again generate the the logits.

1:14:56 And because we now have so many fewer logits to to choose from, this entire process became much much more quicker. And it’s these kinds of tricks that we’re trying to do to make all of these models go even faster. It’s something I think is is really meaningful and I’m super happy to see that there are so many talks during this live stream about what can we do with MTP, what what are variations of it, what’s the research being done. We saw

1:15:26 cool implementations in Video LAN for instance. And it’s these tricks, these these elegant implementations that make make all of existing models go quicker. And that’s what we want in the end, right? We want them go faster. And we can look at the original architecture of the target model, but the draft model does so much of heavy lifting actually. Because so many words and tokens are actually so

1:15:57 so predictable in a way. And the only thing the target model have to do is just catch the direction that we need to go in. And if we do the draft model right, the model can the target model can still be in charge and say, “Okay, this works and this doesn’t.” But in the end, it it validates it and it’s the one that decides, “This is the quality that I want. This is the tokens that I want.” And the draft model, because it’s so quick, speeds up things

1:16:28 quite a lot. And that brings me to the speed ups. We’ve done several tests in different environments. And the funny thing is is that you sometimes even see different speed ups depending on which kernels are implemented for instance. And so you will see a big difference in mixture of experts when it was released on on Mac versus on Linux, for instance. Because when you do mixture of experts, well, it becomes a little bit more tricky to create those draft tokens for a mixture of experts model to route those

1:17:00 um experts and make sure they’re efficiently well cached in a way. And so, you will see all different kinds of speedups across all different kinds of environments and all different kinds of use cases. And here, for instance, well, these are some values, but this batch size one with gamma four, which is the number of candidate tokens that the drafter um suggests. But what if we do eight? What if we do 16? What if we do 20? Well, if you do 20, well, most tokens

1:17:31 will get rejected. What if it were to be incredibly cheap, then in a way you can say, “Why not?” And that that’s a very tough decision often times to make because it depends on the drafter itself. Depends on the use case, like we talked about before. So, is the creative one? Is the coding one? What kind of evaluation have you done beforehand? And so, there are different ways that you can view how it’s being used. But a good default, of course, in the Java four drafters is is what what what

1:18:02 what is implemented across libraries, which is batch size one and gamma four. And that helps already to to speed up your use case. But try it out. Try a different amount and see how it changes because I’ve seen some use cases where eight just works better for me. And sometimes it’s lower. And often times it’s just trying out and see how it works. But especially for production use cases, it helps to not just use it, right the drafters, but also see, “Okay, how can I tweak it a little bit myself to get more speed out of it?”

1:18:35 If you want to learn a little bit more about how they work, we have the the blog release. Um there’s a lot of cool links in there to demos, to explanations, etc., etc. So that’s a nice collection of how they work with Gemma 4. And of course, that’s the thing I like to do, uh trading visual guides. There’s a very large section about what I talked about today inside that visual guide. There’s some snippet of new information there if you’re interested um in it.

1:19:06 Please, please, please take a look. Uh but mostly, thank you for being here. Thank you for being here at those wonderful talks before, at this one. Uh I hope you learned something. >> Thank you so much for coming. >> Thank you so much. It was incredible being here. >> were like >> [laughter] >> you you you would talk about speculators like a fairy tale. >> [laughter] >> I’m sure people have noticed it. >> Uh thank you so much. It’s It was wonderful talking about this and seeing

1:19:38 uh what people do with these kinds of techniques, right? >> Yeah. The illustrator illustrative codes, as I said. >> I can’t use the the illustrated, right? That’s J’s uh domain. I I’m the visual guy. >> [laughter] [snorts] >> Um let’s me see for the questions. Um maybe we can take one and maybe you can answer. There was like less questions and I didn’t know that differs a lot with dense versus MOE. I

1:20:11 today I learned. Um There is one interesting question. Um are we potentially missing critical values if we miss on a cluster assumption? Where is that managed and validated that the info information is reasonable? >> Uh let’s see. So it kind of kind of depends on how you create a cluster, right? It’s It’s not a perfect technique, of course. It’s a proxy still. Right, you create those clusters beforehand.

1:20:42 And how you do that is up to you, but it’s still sort of a one-dimensional cluster that you have because it’s one token belongs to well, essentially one cluster. You can do soft assignments and stuff like that. But I think the implementation here is a hard assignment. And it will miss it. And that’s still the the risk that you have with MTP in general is that sometimes it will miss it. But if all of the other drafters in general are relatively accurate, then that’s okay. That’s kind of the the

1:21:14 balancing that we try to do. So, if it rejects all of them in one pass, but it continues to accept a lot of them in subsequent passes, in a way that’s fine because it’s also not about you know, optimizing every single pass, but the entire sequence of trying to to get it. So, it’s a very cool trick to use. But as all things, it’s not perfect. It will make mistakes. That Frankly, that’s okay.

1:21:45 >> Thanks a lot for joining. So, people are asking if the if this recording will be available. Yes, you can just go to Hugging Face live YouTube channel after this live stream ends, and I will try to link all of the slides inside this description of that live stream. Thanks a lot, Martin and Eldar for joining today. And see you. >> Thank you all. Bye.