Jev Course – Prefill-Only Inference & Decision Models from Scratch

Author: freeCodeCamp.org · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Jev Course – Prefill-Only Inference & Decision Models from Scratch · Channel: freeCodeCamp.org · 1:25:49 · Watch on YouTube

Topics: Model inference and optimization, AI models

Summary

The video explains how to build Jev-style decision models that replace token-by-token text generation with a single forward pass for bounded outputs such as choices, ordinal scores, and yes/no probabilities. It first demonstrates ordinary autoregressive decoding, then compares encoder-only and decoder-only approaches, including how each represents questions and options. The decoder approach can cache a context and reuse it across questions, while the encoder approach uses bidirectional attention but must reread the context for each question. The final sections outline teacher–student training with RLCD, scoring methods, and practical tradeoffs for choosing an implementation.

Key points

Jev avoids autoregressive decoding by producing structured decisions from a prefill-only forward pass, which can make latency and output formats more predictable.
The interfaces discussed are nominal choices, ordinal scores, and binary null/yes-no decisions.
Vanilla LLM decoding repeatedly feeds generated tokens back into the model, using a KV cache to speed up subsequent steps.
An encoder-based approach places the question first, inserts mask tokens before options, and compares question and option representations; it generally requires fine-tuning to yield useful probabilities.
A decoder-based approach can precompute and cache the state, then evaluate multiple questions against it without recomputing the context.
When packing multiple questions for a decoder, the attention mask should let each question see the state and its own tokens, but not other questions.
A first-token decoder implementation can fail when answer options share the same initial token; renaming options or using labels such as A, B, C, and D can avoid this collision.
The training discussion describes teacher–student distillation, GRPO-style perturbations and advantages, and rewards that combine log and spherical scores.
For ordinal score outputs, the ranking probability score (RPS) can account for the ordering of labels.
Encoder and decoder choices involve tradeoffs in context reuse, question cost, context length, bidirectional attention, option length, and model size.

Chapters

Start	Chapter	Summary
00:00	Course overview and Jev basics	The instructor introduces prefill-only decisions, the course plan, and the difference between prefill and autoregressive decoding.
03:07	Decision interfaces and prerequisites	The video distinguishes choice, ordinal score, and binary outputs, then reviews the background and setup expected of viewers.
06:46	Course roadmap and vanilla generation setup	The instructor outlines the course phases and prepares a pretrained decoder model and a structured fraud-triage example.
10:29	Prompt formatting and the decode loop	The example prompt is formatted and tokenized before the model begins generating a JSON response one token at a time.
14:07	KV cache, logits, and token selection	The instructor explains context caching and shows how the final-position logits are converted into a next-token prediction.
19:21	Why avoid token-by-token JSON generation	The generated fraud decision is assembled into JSON, illustrating the extra decoding and post-processing that Jev aims to avoid.
21:30	Encoder-only Jev formulation	The encoder approach places the question before the context and uses mask tokens to represent the available answer options.
25:36	Building encoder inputs and representations	The implementation constructs question, mask, option, and context tokens, then extracts hidden-state representations.
33:27	Scoring encoder options	Question and option vectors are combined to estimate probabilities, with fine-tuning or linear heads proposed to improve them.
40:50	Decoder-only Jev and answer options	The decoder formulation puts the state first, reuses its KV cache, and considers first-token and multiple-choice answer strategies.
46:05	Causal attention for multiple questions	The instructor derives an attention-mask layout in which each question can see the state and itself, but not other questions.
52:19	Implementing the decoder attention mask	The code builds a cached state, segments question tokens, and combines a causal mask with a same-question mask.
1:02:43	Decoder inference and probability outputs	The model uses the cached state and question mask to obtain answer probabilities, then demonstrates reuse across additional questions.
1:09:03	Option collisions and teacher–student training	The instructor reviews ways to handle duplicate initial option tokens and introduces distillation and GRPO-style training concepts.
1:16:19	Reward scores, RPS, and RLCD loss	The video explains log and spherical scores, the ordinal ranking probability score, and how these contribute to the training objective.
1:23:13	Encoder–decoder tradeoffs and wrap-up	The final comparison summarizes when context caching, bidirectional attention, model size, and other engineering constraints favor each approach.

Quotes

Your latency is going to be completely under control to start with but also your output is going to be predictable. — 02:35

So two things one forward pass no decode and predictable outcomes. — 03:07

every time you change the question, the bidirectional states the weights will change and you will have to re-crunch the whole thing over and over again. — 24:35

I can pre-cash the state, and then I can keep on answering questions. — 53:19

Thanks a lot for your time. — 1:25:49

Concepts and entities

Jev, RLCD, prefill, KV cache, GRPO, RPS (Ranking Probability Score), PyTorch, Hugging Face, BERT, Qwen, Clef, Cloudflare, Vivek

Full transcript (original language)

00:00 Learn how to build and master Jev-style decision models from scratch using PyTorch and Hugging Face backbones. By replacing token-by-token auto-regressive decoding with a single forward pre-fill pass, Jev bypasses text generation entirely to deliver bounded low-latency typed decisions, such as choices, ordinal scores, and boolean nulls. You will learn about inference paradigms and explore training workflows

00:30 with RLCD to produce honest, well-calibrated probabilities for real-world software logic. >> Hello and welcome to uh learning Jev from scratch using only PyTorch and bare-bones libraries like Hugging Face. We will use pre-trained models to uh leverage those as a backbone, but we’ll try to learn the first principles of inference and some of the training from scratch on how Jev works, so that you have an idea on uh

01:01 this particular paradigm of how LLMs work. So, we’ll cover Jev and RLCD, reinforcement learning for calibrated decisions, in this video. Hope you enjoy. So, the first thing we want to cover is uh what is Jev, what is RLCD. We’ll probably get to RLCD towards the uh end of this uh particular series, because I really want to cover inference first and the different paradigms under which you can do inference, and then work myself backwards for uh

01:33 understanding the training workflows. So, uh Jev in its essence is very simple. If you understand how an LLM works, an LLM has two stages, uh pre-fill and decode. Uh the pre-fill is the stage where it eats up all of the prompt, is compute-bound in nature, and is about to generate the first token. Decode is generating the answer one by one where if you have 500 words that you put in as a token giving some context for a

02:04 particular task and it’s trying to generate the response the first token is generated and then from there the first token is fed back and 501 more tokens are fed in and the model spits out the 502 token all the 502 tokens generated so far are fed back and the 503 token comes out and so on. What Jev attempts to do is it skips the decode part all together and tries to condense the problem statement in the

02:35 form of one forward pass or one pre-filled statement only. Now this has obviously lots of advantages. Your latency is going to be completely under control to start with but also your output is going to be predictable. It means you specify up front what you want from the model in the form of either options or boolean values and it gives back the exact thing that you’re looking for rather than you

03:07 fumbling with syntax and language and trying to make the large language model predictable. So two things one forward pass no decode and predictable outcomes. Those are the two desirable properties you can say of Jev. Now coming to the interface of Jev you can post questions in three parts. So number one we have the choice

03:38 where you can go all the way from uh almost like asking a multiple choice question. So either you have option A or option B or option C. A score on the other hand does the same thing but is ordinal in nature. So, you can say this is a good example where you have a nominal options and then the quality of that is you have ordinal options. What do you mean by that? So, uh if you if you ask for directions, east, west, north, south, those are nominal in

04:09 nature. East is not greater than west, not greater than south, and so on. So, there’s no order you can maintain within choices. Choices are just uh exist orthogonally. Can’t be put in a certain order. Ordinal on the other hand can be put in a certain order. So, what’s the customer sentiment? See, they’re calm or frustrated or very angry. So, they can be ordered in the order of dissatisfaction, so to speak. So, um the APIs exposed by Jev for choice

04:40 and score are different. And then there’s finally null, which is uh simply binary. So, you ask a yes or no question, you get the corresponding probabilities back. Now, it’s interesting the way it is phrased. So, nominal is slightly different than ordinal is quite different than binary. We will try to come up with a framework that caters to all of them, but we’ll also appreciate and respect the nuances that these three APIs possess.

05:10 Um we’ll try and form a common backbone for everything that we do and from there build on top of it to uh segregate the nuances. So, uh we for our use case will build all three from one softmax per question, but I’ll definitely try and lay out where these can differ. We we come to what are some of the prerequisites for you to adopt this. Uh this is not for everyone. So, if you have a good grasp

05:42 on the concepts of attention, on how KV cache works, on what’s a logic, what’s a softmax, uh you’re fluent with PyTorch, I think this might be for you. If you haven’t done any one of these things, you will find this video a little challenging to follow along. Some things that are good to have but are not mandatory is little knowledge of hugging face. Using a debugger, we’ll use the VS code debugger to go line by line on some of

06:13 the concepts that we’ll try and learn. And then some of the modern aspects of transformers, the encoder, the decoder, and so on is really really good to have. So, these are definitely hard prerequisites. Uh If you’re not aware of these, probably this video is not going to add a lot of value to you. Um And then if you know these as well, it’s great, but it’s not mandatory. All right. What is not mandatory, however, is you do not need to own a GPU. That’s why

06:46 the coding of this whole course is going to be very inference-heavy. We’ll talk a lot about the theory, but we’ll stick to the inference when it comes to implementations. So, you don’t want a GPU to do that. You don’t need any training experience or model training experience or or any kind of mathematics beyond understanding the softmax function. That’s it. So, with that, let’s jump right in. Here’s what we’re going to be doing. We’re going to split this whole thing into three phases. So, phase one will go through vanilla

07:18 generation through LLMs. So, we’ll try and write a prompt that generates a JSON token by token. So, in this phase, we’ll go through the prefill and decode stages together. This will give you a fair intuition of the problem and bring everything into context for us to continue. From there, we will look at an encoder-only implementation. And then from there, we’ll look at the decoder only implementation. Uh Uh we’ll go back to a training run on how it works and then come back and finally do

07:51 a compare and contrast on what approach to take when. Now, before we get into how Jev actually works, I think we’ll need a revisit of the vanilla decoding paradigm or the way they usually uh the way usually an LLM does inference. That will really put things into perspective on where we are headed next and how to reshape this whole thing to point to how Jev wants you to behave. So, we’ve got the tokenizer here and the

08:22 model that we’re loading right off the bat. Again, we’re using pre-trained LLMs. We will be using pre-trained LLMs all throughout the course. Uh in this case, if I take a look at the load function, the model here is a quant 3.6B. Uh it’s tiny enough to fit in most laptops. So, we’ve got the model loaded here. And then we will check out what uh a naive prompt looks like. So, we’ve got the tokenizer here. The context

08:53 looks like this. So, card ending 4412 made nine purchases in 3 minutes across four countries. The card holder called to say the card is still their pocket. Looks like a fraudulent transaction. And the schema that you will be looking for uh looks something like this and this will typically be our uh Pydantic object that you will pass to an LLM provider of sorts like an OpenAI API looking entity. Uh now, the schema contains

09:25 something called priority. So, let’s take a look into that. It’s an enum where the choices are high and low. And then it contains a couple of more items, decision, which is again the choices are fraud or fraud review or clear. And then and escalate. So, you see these three have been carefully chosen to represent the three interfaces that Jev exposes. So, on the one hand you have priority,

09:57 which can be high or low. This is the score API of Jev. Uh there’s the decision, which is again the subjective one. Uh and then from there there’s the boolean. So, these are the three questions that I’ve tried to frame which mimic um all of the interfaces that Jev exposes. And then we’ll try and uh kind of condense this into more simplified examples to see how Jev works later, but let’s first find out what this does. So, we’re going through a naive prompt

10:29 function which simply takes all of this and puts it into a format ready for inference. So, I’m going to skip a bunch of these data handling because it’s not that interesting. So, once this processor comes back we will just take a look at the output and we will be able to clearly see that what were some of the things that were done under the hood

11:00 in this. So, we’ll try and clear that out. So, if I look at the prompt I have tried to kind of put it in a format the way LLM providers like. So, if I were to print this, there’s an IM start token. And then from there the system message starts that you’re a precise data extraction system, blah blah blah. And then there’s a JSON schema that I want you to adhere to. And then the user provides the context. So, uh this is

11:31 what is called the state in Gemma and then this is the user prompt which will be kind of put into state and then the questions will come in later. Yeah? So, in this case in a vanilla LLM decoding all I’m asking for is here is the user prompt, here’s the system prompt followed by the user prompt. Can you please give me the JSON back? Right? So, we even open that curly bracket up so that the LLM can continue decoding accurately.

12:02 All right. So, there was a lot of stuff that we did in terms of changing the data to the right format but it wasn’t really uh, of any interest to us in our pursuit to understanding Jeff so I’ve skipped a bunch of uh, those processings over. Uh, so in this case all we’re doing is uh, tokenizing this whole prompt. So, all you will get is a bunch of um, begins which specify all of the token IDs of the corresponding tokens. All right?

12:34 So, we’ll move ahead. Um, if it has an end of sequence token then great, otherwise we’ll set our own and this will be uh, a non-escaped again JSON curly bracket to signify that we are trying to finish our decoding by a generated JSON. Again, these are all uh, very bookkeeping stuff I would say. Uh, this is where the actual decode begins. So, we’ll try and clear the landing zone to understand this. Let’s take a closer look.

13:06 So, our inputs our tokenized inputs are going through the model which will then spit out an output whose logits we will take, whose past key values uh, we will pluck out and this will become our KV cache and then from there we will look at the last token, go through its vocabulary, and pluck out the the argmax, which is basically the

13:36 highest probability token. Now, if it matches one of our stop sequences, we’ll break. Otherwise, we’ll just keep continuing this, and it’ll be very clear once I kind of uh generate this and show you step-by-step what is happening. So, if we take the model, put it uh put the inputs through the model, and at all times we’re feeding in the cache. So, in this case, the cache is set to none, but then you see the the

14:07 cache gets updated. So, uh what we can do here is if we have the same state or the same context, and we want to answer a bunch of downstream questions, what we can do is cache the state of the context, and then once I ask the first question, I can just retrieve that cache and process it by concatenating the question with that particular context. Um if I have a

14:39 second question to ask, I can just grab the context again from the cache and attach the second question, and do this so on and so forth. So, I’m just trying to maintain a cache, which is just a best practice. All right. So, from there, uh the logits come out. So, just for perspective, I had uh let me take a look to the inputs. So, we have the input IDs that will give me an idea of the number of tokens.

15:12 So, if I take the length of this, I get one, which is because it’s a nested JSON. So order it’s a nested array, I’m sorry. So there are about 130 tokens in total in this whole thing right now. All right. So from there once the outcome is out now you will see that if I access

15:43 the logits then I will have 130 outputs and each token position will have the vocab size number of options to choose from. So uh the way the transformer works is if you put five tokens in, you will get five predictions out. Each prediction will contain the entire probability distribution of the vocabulary in question. So if you’re only looking for the last token, what you do is you throw

16:14 everything away apart from the last prediction and then use that probability distribution to pluck the argmax, which is exactly what we’re doing here. So we’re updating the cache. Now from those logits we take the first dimensions. So since we’re only processing one batch if we just take one the first one and check out its shape, you will notice that the one in front is gone. And what we’re left here is the 130 by

16:46 the 152k uh stuff that we want to look at. And then from there if I take the last element of this I’m essentially left with the probability distribution of the predictions of the last token. So if I take a look at the shape of this, then I have 152k outputs that I have to deal with. Now from there all I want to do is convert them to float put them through a softmax, and this is just dimension minus one.

17:17 So, uh it does a soft max over all of these numbers. And then I’m going to take the arg max. So, if I really take this and do them one by one myself, so you will get a tensor of 130 uh rather 152k almost um probabilities. And then if you take out the the arg max of them, you get the token ID of the token that has the maximum

17:48 probability. So, uh arg max give you the position, which is nothing but the token ID. So, I can actually decode this and get my answer. Let’s see. This is 330. So, this is the token whose uh which is So, this is the token that was predicted in this loop. So, if I just let this run for a couple of iterations and again,

18:19 uh check out the token, the freshly minted token that was generated, you will see that uh it will keep changing over time. So, now it becomes priority. So, then I think we’ll probably have this thing um the the inverted comma closes along with the colon sign. And if we keep doing this for a few iterations,

18:50 then you will see that the accumulation that is happening in this grown variable will reflect the answer that we’re looking for. So, this is exactly why you see chat GPT, the web interface, give you tokens one by one. This is because uh it’s looping through all of the previous context and generating these tokens one by one. So, it reckons that the priority of this particular ticket or particular context is high.

19:21 And then, if we keep taking a look while iterating through a couple of turns of this, then you will see that it will subsequently reach a place where it has almost generated the entire JSON. So, if I So, if I take a look at this again, uh it’s done predicting the second one. So, it reckons it’s a fraud. And then, we’re left with escalate.

19:54 has more tokens to generate. And if we come towards the end after it has uh finished its loop, then we will be able to see the final text that it predicted after all of the iterations of generating these uh tokens one by one. So, if we take a look at the text here, you will see that it has returned our final JSON. Uh let’s actually print that to see it very clearly. So, we’ll

20:25 we’ll print the text so that it uh pretty prints it. So, then you can see that uh starting from this particular inverted comma to the word priority to then again choosing another token for this particular text and then generating token by token by token took a lot of time, which uh could have completely been avoided if we were smart about formulating our problem statement in a way that only let us um make predictable assumptions without

20:58 getting into this vicious decoding cycle, at least for this problem statement. Right. So, there are a bunch of this uh bookkeeping stuff that takes care of uh trying to make sure that this ended up generating a JSON by looking at the escape characters, looking at the start, looking at the end, etc. etc. And then finally, once that post-processing is over, you’re left with an answer. Now, we saw how a decoder-only model was used

21:30 to generate a response. We’ll begin now with one of the first formulations of our generative model. And this will be through the encoder backbone. So, what we’ll do is we’ll try and leverage an encoder model like BERT or any other variant of it in order to try and reformulate the use case so that we can avoid the decoding part altogether and still see if we can get

22:01 all we want to get back. So, first things first, we’ll take an encoder-only model. Uh we won’t be doing any training. And we’ll try and see how well we can retrofit for this particular use case. And if not, what are some of the training approaches that can be taken. All right. So, the first thing first, you’ll see that we will encode the question here first.

22:31 Now, this is a very unique approach. Usually, you provide the context, then you ask the question. Here, we are doing the opposite. We are starting with the question and then providing the context over here. Now, this approach is because attention, the way typically works, is it places a lot of importance uh to the beginning of the tokens in the attention block. So, we want to make sure we want to put that in. Uh experiments, I think, have been done with the reverse, and this has just ended up performing better. So, we’ll

23:02 stick to this implementation. Now, once the question is asked, you have all of these options. So, in our case, it was either high or low or fraud or review or even yes or no. In this case, before every option, we will have a mask token. This is a special type of token that we will inject. And we will kind of sequence the whole thing. We’ll orchestrate the whole thing. And then, we’ll use this prediction effectively to gauge what option we can

23:34 predict. So, if you have option B, we will inject a mask token against or just before option B. If you have an option C, we’ll do a mask token before that and so on. And then finally, we will have the entire context over here. So, once you see this, we are we are going to have to do this for every question. So, if you have a really long context here, um every time you ask a new question, the state has to be re-fed in this whole

24:04 thing. So, you can ask, uh it’s an encoder model. Why can’t the state come first? Why can’t I cache it? And then, why can’t I just swap the questions one after the other? Well, here’s the problem. Uh encoders are bidirectional in nature. So, every time you actually change the question, you will end up completely changing the attention weights in every

24:35 position of your attention matrix. Because of which, even if you flip this order around and have the state come first and the question come last, you will still every time you change the question, the bidirectional states the weights will change and you will have to re-crunch the whole thing over and over again. Um I hope you can see this. It’s a very simple interface, but then you can clearly see that for every question you ask, you will have to keep appending the state over and over again. So, you if you have a lot of tokens

25:06 and a lot of questions to ask, this might not really be that scalable. But, given that it’s bidirectional attention, it will give you the maximum bang for your buck when it comes to every model parameter. Yeah? So, without further ado, we’ll dive into the implementation on how we’ve got a pure encoding.py. And we’ll set a breakpoint around here, and we’ll take this for a spin.

25:36 Uh what we are interested is that we’ll have a state. We’ll have two questions just to see how it interacts with the multi-question setup. And then from there, we will see that how we are embedding the question and the mask and the context all in one. Now, one thing I have actively avoided doing here is I’ve not tried to batch the questions together, although you can. Uh I’ve really gone old school on doing for loops so that we can pick it apart and see how this is working. This is not the

26:08 most efficient implementation. All right. So, first of all, um there’s this function called build, which takes in the state, which is nothing but the context. So, we’ll take a quick look. So, this is our context. Ticket 8812, charged twice for the same order, no reply in 4 days, cancel it and give me my money back, or I’m disputing with my bank. All right. So, the state along with the

26:40 instructions, so each instruction will be a question. That’s how we are formulating it. So, this for loop is tending to the first question right now. Um the moment it loops again, there’ll be a second question. And then there are options here. Again, this question is yes or no. There can be multiple options as well. So, with this setup, let us get inside the build function to check what is going on.

27:11 Now, it’s really easy. We take the instructions, which is the question, and then encode it first. So, there we go. We get the token IDs of these question back. And we are kind of doing it in a way that we put a classification token first and then the question and then a separator token. So, uh, once we do this, we will actually keep a tab on these separators and delimiters that we’re injecting so that we can pluck out what we want to pluck out in pursuit of our,

27:43 uh, finding the answer. So, here we have the span, the span of the question. And then we have markers. Those are nothing but the options. So, we already know for the first question the options are no and then yes. So, uh, we’re injecting where I’m going to mark my mask token so that I can keep track during inference. You’ll see all of this in a minute if this is not clear.

28:15 All right. So, we’ve got the markers appended and appended to the IDs. Now, if we, I think, if we track the IDs, we should be good. So, IDs was about just having the question, but then notice we’ve uh, put in the classification, uh, token and then a separator token once the question finishes. The mask token that we will observe and we will

28:47 inject before every option so that we can track the masked tokens probabilities and do some things to it in order to gauge the strength of that option or the probability or the propensity towards that option for this question. Yeah. So, once we loop through this again I’m expecting there’ll be one more masked token followed by the yes option. So, let’s see that in action. Right. Once this is added in

29:19 you will see that there’s another mask now catering to the yes option. Uh the options are all exhausted. The loop won’t run anymore. And then finally uh the IDs are constructed and returned. So, if I do a final check then you have the classification separator followed by the question then the separator the mask for the first option the second mask for the second option then a separator and then finally I dump the entire context followed by again a separator. So, these are uh

29:51 very encoder lingo BERT-style MLM-style masked language modeling-style training paradigms. Uh and this is used in inference all the time if you’re using BERT. This kind of uh ordering of the tokens so we can differentiate between what to look at and what not. All right. Uh so, that’s the output of our build function. And then we get back and this is literally the LLM call. Uh so, we pass all of our input IDs. Now,

30:22 the input ID here my new contains the question, the option, the state, everything. And then what I’m doing is I’m plucking out the last hidden state and then moving that into the CPU. So, we’ll take a look at it in what second. So, my model prediction is over. Here’s what it looks like. So, the hidden shape is 54 cross 768

30:52 in nature. So, this is literally um each token has its own embedding that we’ve gotten. So, if you look at the number of tokens that we have was 54. So, the hidden state returns 54 embeddings for each of those tokens. Now, uh if we ask for the last hidden state, um what it will give us actually, let’s run

31:24 this whole thing. So, if I run through this, it’ll do another model prediction. So, outcomes um 1 54 768. Now, if I ask for the first piece, then it will get rid of that batch dimensionality and just give me 54 by 768. So, if I check the shape of that, so that’s 54 by 768. And then, I’m converting this to float. And now, in the next step, what I’m

31:55 interested in doing, I’m interested in plucking out just the query. So, again, the spans and all of these variables have been pre-made. We scripted a bunch of them to save on time. Let’s look at them now. So, if you look at span, it’s a tuple. The first element of span denotes the starting token or the starting position of the first question. And the nine here represents the end

32:26 position of the first question. So, in fact, literally, if I do one to nine, and then try and decode it, I will get back the question itself. Is the customer asking for money back? Yeah? So, why did we start from one here? If I had start from zero, then I would have encountered the classification token, which I want to avoid. The question itself does not uh contain the token. Right? So, I will take

32:57 the hidden states of the questions themselves. And if I look at it, um there were eight tokens in the question, so there’s 768, and I’m mean pooling them. So, I’m trying to come up with a 768 vector representation of the query itself. That is how you interpret this. Now, if I take this and I take the mean,

33:27 you will see that I will be getting 768 numbers. I’ll repeat this. This is the best representation of the question that we have from the model. Because we’re taking all the tokens in the query or the question, and then have averaged out their embeddings to get the best representation of the question itself. >> [clears throat] >> All right. From there, we have got our markers, so we know for a fact that uh in the 10th place, again, we can

33:59 check this. We’ve got the mask token, and if we look at a token before this, then you will see that uh or rather or rather, if we look at a token after this, you will see that it is succeeded by the option that we’re interested in. And now, if I take a look at the the 12th

34:31 marker, then you will see that it is the marker marked for the yes option. So, we’ve got two options, one for each of um one mask for each of the options. And then uh we take up and find the hidden states of those as well. Again, we’ll get those two embedding representations of the two options. And now here is where

35:01 our implementation is really going to be the vanilla part, and here is where a lot of variation happens in implementation. So, let’s try and understand this better. So, we’ve got the question here. And then um the mask token, then the option one, then another mask token, then the option two, and then so on and so forth till a separator comes along and gives me the entire context or the

35:34 state. And then finally, gives a prediction. So, what we are trying to do here now is that each of these token generates a 768 uh embedding vector. Now, the question, if I take the span of the questions, the question itself will span a few tokens, and I average them out. Then this will give me a clean representation of 768 embeddings. Now, this mask token

36:06 will also have a representation, so M1, and this will also have another representation, like so. So, you see the options span 2 cross 768, and then the query spans 768 cross 1. So, this is the option. This is the query. Now, one way to look at it is that I can make these two interact with each

36:36 other. So, I can literally take a dot product and I’ll be left with two scalars and I can do something on top of these scalars, even take a simple soft max on top of these scalars and deem them probabilities. This is literally the simplest way to formulate a Jeff problem. Um no matter you have a choice, a null, or a score, you take all of these tokens, make them interact with each other. It’s

37:07 bi-directional attention anyway, and then from there take the query vector, make a representation of it, take the masked option vectors, make them interact, and you can interpret the output as probabilities. Now, obviously, we are taking an out-of-the-box backbone with no training whatsoever. Uh this won’t work, but this serves as the bedrock. Now, what else can you do with it? Now, you’ve got option one, uh linear head.

37:41 What I mean by that is instead of um taking these 768 vectors each of options, I can make them go through a linear head each and outcomes a scalar. So, that linear head literally can go 768 and outcomes one. So, if an option vector goes in, a probability, the corresponding probability comes out. So, this option

38:13 um matrix here can literally go through a linear head and then outcome two scalars that represent the corresponding probabilities of each option. Second, I can do a full-scale lot of fine-tuning. Um and I can set an objective either by taking the way I formulated the problem where the option was trying to interact with the query, or I can fit in the linear heads and then have a full lot of fine-tuning. Option three, I do a full

38:45 backbone retraining. Now, it turns out in practice what has happened is the embeddings that are learned, all of these 768 uh representations, they’re actually quite high-quality and high-fidelity, but they don’t know that they have to be outputted in a probability sort of a way. So, uh whatever number you get out of these won’t make sense, but all the signals in the world are already present. So, it really depends on how complex your task is, on how much data

39:17 you have, or you’re willing to put in and invest on it. Um you choose one of these options. In its most vanilla state, you can get probabilities, but it won’t take you much. Uh you may have to put in a linear head and fine-tune just that linear head. You may have to do a full lot of fine-tuning or a full backbone retraining. So, with that perspective, we come back here. Now, we’ve got the option vectors. Remember, this is 2 cross 768. And then, we’ve got the query vector, which is 768 cross 1. So, you make them interact and normalize them.

39:47 You actually put them through the softmax and you get probabilities. So, here in this case, if I take a peek, you will see that two scalars came out. Uh and these can be looked upon as the probabilities to answer that particular question. Now, I know they’re not very accurate. This is supposed to be a yes, whereas it’s only emanating 30%, but mind you, uh the embedding signals are very high-quality. This interaction needs a bit of a getting used to. Needs a bit of

40:19 a slight fine-tuning, which is where our three options come in. Yeah? So, with this, I think uh we’ve already completed our encoder representation of how Jeff works. Um I hope it gave you a sense of intuition on what all you can do, and what all you have to do in order to make this work. Hope this helps. All right. Let’s actually check out what happens in the decoder world. Uh now, it turns out you can also use a decoder in order to formulate the

40:50 problem in a way that it only takes you one forward pass, only the pre-fill stage, no decode stage, in order to achieve all of Jeff’s objectives. And here’s how it works. We are going to pack all of the state upfront. This is exactly the opposite of what we did with encoders, um because this time this can be cached. So, if a state, and since this is the causal mask, if the state has a KV cache existing,

41:21 I can just keep swapping the questions here and reusing the KV cache just as well. Now, the other thing that I can do is if I have a large enough context length, I can actually pack as many questions and answer options that I can over here. So, imagine uh you have 32K worth of context, and then this only consumes 1K, the state. Now, just imagine you have 31K context to

41:51 fill out as many questions and options as possible. Uh this is kind of evident that Jeff has taken a decoder approach, uh because it supports up to 5,000 questions over a single state. So, my best guess is they might be doing decoding, but uh again, they haven’t released anything, so can never be sure. Uh the first token or the the approach that we will be taking here is that we will formulate a problem in the way that here’s the context,

42:22 here’s the question, and we’ll just leave it at a colon. What this does is it actually sets up the LLM almost as an auto completion task. And the moment it spits out a token, we will take a look at that token, and it will obviously have an entire probability distribution of its vocab size. We will actually focus on only the tokens that are there in our options,

42:54 pre-trained or rather pre-built. So, if a choice has only got yes and no, we will only look at the probabilities of the tokens of yes and no in that first token decoding step. And we will ignore all of the rest, and we will treat them as probabilities and return them. Now, instead of that, you can also formulate the problem in this way, where uh you rewire the problem to almost like a multiple choice question. You have have

43:26 options A, B, C, and D, and then you prompt the language model to answer what the answer is and nothing else. Either it’s A, B, C, or D. Now, based on what letter it chooses, uh you spit out the probability accordingly. Now, these but two approaches do not require training. Everything else does. So, uh we will stick with this part, and this can fairly be extrapolated because again, uh it’s just prompting the model to spit

43:58 out one character, uh which are one of the options that you have. And then you can have again, a decision to option uh pointer head, so it goes through a linear layer again in order to do an interpretable probability distribution. This is what uh Jeff has implemented. This is an open-source alternative to Jeff, and then Cloudflare came up with Clef, which is basically a pooled span implementation of the whole option plus uh a particular linear head.

44:30 So, uh it takes into account the entire option for those questions, so it can never really go wrong in the case of duplicates. So, just to be clear on what I meant by that last part, uh is this legit can be, let’s say, a question. I just made it up. Uh option one, fraud, and then option two can be fraud review.

45:03 Right? So, if you have and it’s uh a perfectly formulated Jeff-style task, where you have option one and option two, notice that the first token are exactly the same. Now, this is where our simple implementation will go for a toss. Uh it won’t know the difference between these two, because the first tokens are literally the same, and it’ll probably spit out 50/50. So, what I’ve done in my implementation is handle it so that if there are duplicate tokens, or if there are the same tokens, we’ll just throw an error prompting the user to change the

45:34 way they formulate the option. So, uh this can literally be just review instead of fraud review. And then uh that thing comes for free, right? So, uh again, let us lots does not have that problem. If you just call it A and B, and then ask the LLM to give you one of these letters, I think it should be fine. Uh but then uh pooled span especially takes that to the extreme and avoids it completely, even if there are completely duplicate options, uh this particular implementation has

46:05 the scope to disambiguate it. We are again not going to go through any of these three implementations. We’re only going to stick to the first token task and see mainly focus on how the cache can be reused and how the question swapping occurs. This in itself is going to be a challenge. Before we dive into the code, I want you to have an understanding of how we are going to formulate this causal mask block because this is really going to be important for you to visualize in order to understand how we can pack multiple

46:35 questions in a single sequence. So, let’s say we have a question. Um uh we have some context. Um My name is Vivek. Now, from there the question becomes Oh, what is my name? And then maybe uh What is

47:07 or rather we formulate it in this way. Uh the option can be uh Vivek or something else. And What is my surname? And can be anything. So, option A and option B. Okay. So, the way we’ll formulate the problem is this. It is causal mask after all.

47:38 So, uh token one, token two, and so on. So, if you really visualize the attention matrix over here like so. The first token can only attend to itself and it cannot attend to anything else. Now, the second token can attend to itself and the previous token and so on. This continues, right? So, uh at every step each token can only look back and not look in front.

48:09 Now, this in the context of when you have multiple questions and you want to pack all of them in a certain context, gets a little tricky. All right. So, let’s say uh the question is three tokens and let’s say for arguments uh rather the state the state is three tokens. For arguments sake, you can ask one question along with their options in the next two tokens.

48:40 And question two also uh takes two tokens. Again, for so that we can actually see the full-blown matrix. So, what will happen is you’ve got uh state one, state two, state three, and then Q1, Q 11 12, and then Q21, Q22. And then the same thing goes here, S1, S2,

49:11 S3, Q11, Q12, Q21, and Q 22. Now, the way you want to pack this is that this will be definitely uh vanilla attention as you know it, typical masked attention. Causal mask, yeah? Now, this question should be able to see the entire context. So, I’ve got no problem continuing the causal mask that we are building over here, right?

49:45 We are good so far. Now, what should happen to question two? Now, ideally, question two should have all the access to look at the state, but why should question two have any access to look at any content of question one? I mean, it is theoretically possible to expose question two to questions question one’s context, but think about what you’re doing here. You’re actually poisoning the context with a question you’re not even supposed to know or look at.

50:16 Uh what this will introduce is that if you ask questions in a particular order, you will get different answers. So, it won’t be very reproducible on your domain. Uh very interestingly, as far as I’ve studied Cloudflare’s clef, has done this implementation where they haven’t masked, and it uh just doesn’t make any sense to me. Uh because the once you mess up the order of the questions, you will get different

50:46 outputs, I’m sure. So, if you look at this, uh question one, or rather question two, should have all the access to the previous state, should have no access to the first question, and then should have access to its own token, of course. So, it has access to the state, has no access to the question, has access to itself. And this is how

51:17 we’re going to have to So, the state will be its own block, and then when it comes to the question, the question will have access to its own block and the state. And the next question will have access to its own block and the state. And this will be completely masked out. So, constructing this mask will require some array gymnastics, which we’ll soon look at. And that’s why I wanted to really focus on

51:48 uh the first principles and the core aspects of constructing this rather than uh looking into a varied set of implementations and confusing ourselves. Once you understand this fundamental, I think the different varieties of implementation, the different uh tweaks in the approaches that the different providers have taken will be easily accessible to you. Without further ado, let’s dive right in to the implementation. So, we’ve got the pure decoding script

52:19 here. And since we have some of the perspective on the theory, we’ll quickly look at the state. So, this is the context you’ll be provided. So, “Just twice want my money back.” And the questions are essentially two. If I look at the first, is the customer asking for a money back, yes or no? And which team owns this? Is it the billing team or the technical team? These are the two questions we’re interested in asking. Uh again, this can easily be extrapolated to a score

52:49 question as well. I’ve only included null and choices in this matter. Yeah. So, the first things first, we’ve loaded our model. This is a decoder-only model, so we’ll use the exact same model as we did in the vanilla decoding stage. So, this model is going to be a Qwen 3.6 billion parameter model. Uh and then from there, we are providing the token, the model, the state so that we can get our cash back. And this is one of the most desirable properties of

53:19 decoder-only. I can pre-cash the state, and then I can keep on answering questions. Either I can pack those questions together, or I can just go on a loop answering them one by one. Yeah. So, we’ll go slightly inside the prefill stage to take a look at what is going on. And then you’ll be able to see that the entire state is getting tokenized like so. Uh And then I’m actually doing a model inference over here. And output with the past key values so

53:50 that I can get my KV cache for this particular state. So again, if I take a look at the IDs that went in, there are 38 tokens. So the KV cache uh typically as far as my understanding goes, it will have some layers, some KV heads. So all of that put together and of course uh key values so there’ll be two. So two cross the number of layers plus the number of times the number of heads times the dimensionality uh of the

54:21 embedding. All of that will matter when it comes to plucking out the KV cache. So uh the dimensionality, if I take a look Oops. Uh this is Let me take a look at the length. So there are 28 layers in this model. It tells us something. All right. And then each layer has this uh KV cache.

54:52 Or rather, how do we access the first value? I think we do do it by the layers property. This is the first layer. And then here I can take a look and hunt for that particular uh So there you go. So we’ve got the keys and the values as tensors here. So if I take a look at this shape I will have the batch size. So

55:23 this has eight KV heads, 38 tokens, and 128 dimensions. And there are a total of 28 layers. So this times 28 times two because this is just the keys, we also have values. All right. So, from here uh we pre-cash our state, so to speak, and now we are ready to answer questions. So, we go from Let me run it till here, and then we’ll

55:53 go deeper on this particular function, answers. Right, so uh we’ve dealt with the state or the context. Now, we are going to deal with the questions. So, we’ve got the questions here. It’s a list of um tuples. Each tuple is a question and a list of answers. Or rather, a list of options. All right.

56:24 So, this is where we begin the whole exercise of trying to make sure the right tokens are um getting served by the right tokens. This whole question two not being able to see question one and so on. So, we’re constructing this question mask. Let’s take a quick look. So, you’ve got the width of the segments. Uh this will contain the boundaries of the questions themselves. So, if you see um

56:56 if I have a total of 35 tokens that comprise the questions. So, the state has been cashed already. We don’t need to worry about it. These are the 35 tokens that contain the two questions along with their options. And we’ve seen that the roughly half of them are question one and the other half is uh question two. So, the segments denote the actual index of the question.

57:26 Right. Convert them into a torch tensor. Now, uh we construct a basic causal mask with the A range and then we make it uh, greater than equal to its own transpose so that we can come up with our causal mask. So, if we take a quick look, the causal mask is simply uh, the first element is true, the first two elements of the second row is true, the first three elements of the third row is true, and so on. This is the vanilla causal mask.

57:59 Uh, we’ll need to do more stuff, as we’ll see, in order to get that uh, whole question-to-question abstraction that we want to achieve. So, whether it is the same question or not is dictated by the segments, and we’ve got the segments uh, in a torch tensor called seg. So, we’re going to do the same thing, uh, but with an equality clause to its transpose, and in this way, I’ll be able to gauge whether uh, two tokens are part of the

58:29 same question or not. So, if I just go through uh, the same question tensor that I have, let’s see what that spits out. So, you see, you will see that the first few tokens here are all true with each other, and the last set of tokens here are all true with each other. So, there’s clear segmentation of these are the tokens of question one, these are the tokens of question two. So, now, if I take that and make it interact with

59:00 the causal mask that we constructed in the previous steps, some interesting properties will come out. So, we’ll just wait for that to happen. Yeah? So, in the next step itself, the visible should be the concatenation of torch.ones, the width of state length, along with the interaction of the same question and the causal mask that we’ve constructed. So, this will give us that uh, the desirable state of what we wanted to achieve. And this I think loops through for

59:30 questions. Okay, so we’ve got the visible tensor with us. Let’s take a deeper look. So, we had the causal mask, we had the same question mask, we made them interact with each other to come up with this. All right. So, we can see that uh I think this entire shape is 35 cross 73. So, there are 73 tokens in total which constitutes the state and all the

1:00:03 questions. These have been packed together. Now, since the first few tokens have been cached in already, I just need to care about the 35 tokens that comprise the question. So, if I take a look here, what I have is this piece of the puzzle entirely. So, I have all the questions with me. I have this part. Hence, this is 35 cross 73. I don’t have the top part because I have already cached that in. So, this part is already

1:00:34 cached in. I’m just interested in this entire part. And hence, I have this piece with Hope that is clear. So, the moment you see visible, you will see that they resemble the state. The first part is the state where everything is true. Again, if you look at this part, all of it is dense true. And then the only interesting decoding

1:01:06 part starts once it crosses over the number of states over to this. So, if I were to really slice and dice this to ignore maybe the first uh 38 tensors which contain the context. So, let’s actually do that. So, from the 38th token onwards, you will already see that we’ve got this causal mask running because we’ve exhausted the state which had 38 tokens in it. And

1:01:39 then from there you will see that uh everything in the bottom left is false because these are the tokens of the first question and we are taking care of the second question now. And then everything is just causal for question two. Everything is causal for question one here. Everything else is false on the top right. Everything else is false in the bottom left. So, we’re trying to maintain that mask uh that we saw here. That neat mask where uh here everything has to be turned on well. Everything here has to be turned off. And everything here has also got to

1:02:10 be turned off and so on, right? So, only we look at um questions looking at their state. That’s what we want to achieve. So, hopefully this was clear on how we are constructing this mask. It’s a little different from the vanilla causal attention mask. So, that’s why it had to be kind of deep dived into. But once we get the mask back, everything else remains the same. We’re going to go through the same journey that we go through in a vanilla decoding state where we only just do one

1:02:43 predictions and one prediction only. So, we’ve got all of these segmentations and all to boot. Um we’re going to pass that attention mask that we constructed along with the IDs that contain all of the state, the question, and so on. We’re going to get our uh we’re going to pass our uh KV cache here because we’ve already done that for question. Um we’ll pass in the position IDs here as well. And

1:03:13 if you take a look here, you will see that it spans all the way from 38 to 53. So again, keep in mind uh while we are feeding it a fresh tensor, a cache of 38 tokens have already been fed. So the correct positions needed to be uh filled in so that it can uh calculate the position embeddings and rope and everything properly. Yeah? So we get the logits.

1:03:48 And we’ll just run it till here. So we get the logits for this. And if you take a look at the shape of the logits, it’s typically what an LLM should do is that the number of tokens that you passed, it will give you the exact same thing back, but along with um for each token position, there’ll be a vocab size uh which will contain the distribution of the probabilities at each position. Now, we will only be interested in the last token because we are assuming that the

1:04:19 last token itself contains our non-duplicate answer. So we’re going to uh dig down here. So for every instruction choices decision option IDs in questions, decisions, and candidates, uh let’s actually take a quick look on what these arrays are doing. Questions is simply the text of the questions. You’ve seen this before. Uh decisions, 18 and 34, is actually uh the

1:04:51 end decision point of each question. So um if I take it take a look at the first 18, uh Uh, you will see that it contains the question along with the options. And if you take a look at everything else from there,

1:05:23 you will see that you’ve got the second question as well as its options. So, decisions are being taken based on those questions and their options. And of course, we have the the candidates. And these are uh, the token IDs for yes or no and billing and technical. So, we’ll take a quick look here as well to the candidates so that we can assign probability. So, this one is a yes, this one is a no, this one is a yes. So, 902 is a no, 9834 is a yes, 30531

1:05:55 is billing, and uh, 10916 is technical. So, that’s how it’s um, shaping out to be. So, we’re going to enumerate over this zip of questions, decisions, and candidates. Now, obviously, we only care about the decision points. So, we’ll go right there. This is where question one ends, and it was able to look at all of the state before. And this is where, in the next loop, question two will end, and it will also have looked at all of the state and

1:06:27 context before that. So, that’s the last token that I care about, whose logits I’m going to grab and convert them into probabilities. And they will represent in my case, the corresponding probabilities of the options themselves. So, let’s take a look at them. So, we’ve got the probs here. Quick look. So, we’ve got, for the first question, 95% confidence that

1:06:57 the customer is asking for his money back. So, you see, with the decode only, it works right out the bat. Uh, unlike embedding, switch needs some sort of fine tuning. All right. So, for the second question, we’ll go through the same journey where uh which team owns this? Is it billing or technical? And then the probabilities we’ll see here for those two options. And if there are three options, three probabilities come out. This is not a problem. So, 97% confidence that it is uh of the billing

1:07:29 team. And about 2 and 1/2% of the rest, which means it’s from the technical team confidence level, right? Uh so, that we go from here to the end. And practically once we’ve reached this stage, uh you will see that I’ve got another line of code here which says, “Answer, how angry is the customer?” Calm or furious. And this is a score-style question that you can run it

1:08:00 yourself. I didn’t bother running it. But the magic of the decoder-only style I wanted to show you is that this does not require recomputation of the context or the state. It has been precomputed. So, you can keep adding more questions here. And changing them, changing the options, none of that matters, none of that will change. Uh it will all be the same. So, the the state will never change. It has been cached in. The questions

1:08:32 you can uh in fact have a follow-up with the user uh as well instead of batching them up. So, I hope that gives you a fair understanding of how the decoder-only JAF inferencing works. Uh as a bonus, we will try and cover how the training works as well. Uh we will not be going through code or going through um any of the data sets and stuff like that. We will, however, formulate the training objective. We will look at the loss function and how it’s computed and

1:09:03 what are some of the varieties there and we’ll you’ll really get a flavor of RLCD in theory. So before we dive into the training part, I really wanted to tell you what some of the shortcomings of the method I show you are and how you can kind of avoid them easily. So when two answers start the same way and I’ve already shown this to you before if two answers start from the same token our implementation will fall apart. Now

1:09:34 a free thing you can do in order to still retrofit this to the simple vanilla approach that we have taken is simply rename them. If not we need to score every token and this will take us one extra pass. So instead of just one decode step, we might need to go through multiple decode steps and then do a do a mean pulled scoring strategy. Or you can really [clears throat]

1:10:04 condense every option and encode them into letters like A B C D just like MCQ questions are formulated and answered. This will also make the collision go away. And of course you’ve seen this before with encoders you have the mask token stapled just before an option. Because it’s positional in nature, it never confuses an encoder. Duplicate tokens in the choices are only a decoder problem, not an encoder

1:10:34 problem. Yeah. So with that we’ll get to the training part. We will go through examples of how training works, but we won’t be actually showing you a training run and actually coming out with a model. Um leaving that as an exercise. Some of the extra scripts or bonus scripts that I will share as part of the GitHub will contain those training steps, but also they will be with synthetic data. So, it will be of very little real world use.

1:11:07 But I’m sure you can take them as template and reconstruct it for your own use case. So, with that, we’ll look at one training step worked out. So, the way these trainings are formulated is through teacher student distillation. That’s pretty much it. So, either you have pre-labeled data or you have a teacher model that has the ability to give you some sort of probabilities. So, let’s say

1:11:38 for our previous question, is this a billing problem or a technical problem or account problem or whatever, you have two option three options, and the teacher model says 0.7, 0.2, and 0.1. This is the probability distribution. Now, our scores, you can think of them as logits or scalars coming out of the linear head or whatnot. So, they haven’t been soft maxed yet. These are the scores. The raw scores are +1, +0.8, and +0.20.

1:12:12 Now, our answer our probabilities are 0.44, 0.36, and 0.19. Now, notice that they maintain rank order, but still, I would like our answer to be closer to that of the teacher. So, what do we do now? Now, we use this paradigm called GRPO, group relative policy optimization. It’s a concept in reinforcement learning. Um

1:12:43 if you haven’t gone through GRPO before, you should probably get an intuition into it. I think it will dilute our focus by a lot if we try and explain what a GRPO does, but in essence it is trying to follow multiple trajectories, taking their average, and if a trajectory follows a good path or generates a really high fidelity answer, we subtract the mean from them, and what’s left is something called advantage. And we use that advantage to construct a loss, and then back

1:13:13 propagate to ensure that the LLM spits out the good answers more than the bad ones. That’s reinforcement learning in a nutshell. Uh in this case, we don’t have multiple trajectories because, again, we’ve constrained our model to generate certain token probabilities only. So, we’ll have to manufacture them out of thin air. So, what we do is we take these scores. So, we’ve got these scores here. So, we’ve got 1.8.2, and we add jitters. We add random

1:13:45 perturbations to these scores. And let’s say trajectory one, which is certain kind of random perturbation, yields these scores. Uh on the second leg, we add another set of random perturbation, and it does this, this, and this. So, instead of the original scores, we’ve morphed it and added added random jitters to come up with these particular scores. Now, if we convert them into probabilities, they look something like this. Now,

1:14:17 the first trajectory is something that I like because it moved towards the direction of the teacher while still maintaining rank order. So, I would grade this higher than the rest, and then since uh each of these things receive a grading, their average will be subtracted out, and you will see that this has positive advantage. So, this is literally what I was trying to explain is the advantage. So, anything having a positive advantage will have a lesser loss and will be

1:14:47 favored to more compared to the others. So, this will nudge our student model towards the right direction, towards uh mimicking the distribution of the teacher, so to speak. Yeah? So, just to show you some of the calculation. So, uh these are the rewards that we got. So, -0.33, -0.52, -0.5241, -0.4154, and they average out to become

1:15:18 -0.4498. Now, if I take the reward and subtract the mean from it, so I’m essentially doing -0.3380 -0.4498 divided by 0.0901, is again the standard deviation of this whole distribution, and out comes the answer, which is I’m literally calling it the advantage. So, these two trajectories have a positive advantage, while these two have a negative advantage, and if you look at their answers themselves, you will see why they have positive advantage intuitively speaking. So, we had a 0.44, which is our baseline. Uh it bumped it up ever so slightly, while decreasing this, while further decreasing this, or maybe uh this kind of increased a little, which is So, there is an overall

1:15:48 advantage of gravitating towards this trajectory. Now, this one has maintains the rank order beautifully, and also gravitates more towards the teacher model, hence it just has more advantage. So, the the scoring mechanism has been come up in this way that uh with these trajectories, the better answers have more advantage, which then affect the loss more. Yeah? So, the one last question that is remaining here is if you look at their answers themselves, you will see why they have positive advantage intuitively speaking. So, we had a 0.44, which is our baseline. Uh it bumped it up ever so slightly, while decreasing this, while further decreasing this, or maybe uh this kind of increased a little, which is So, there is an overall advantage of

1:16:19 gravitating towards this trajectory. Now, this one has maintains the rank order beautifully, and also gravitates more towards the teacher model, hence it just has more advantage. So, the the scoring mechanism has been come up in this way that uh with these trajectories, the better answers have more advantage, which then affect the loss more. Yeah? So, here is where did we magically get these scores? So, let’s explore one of them on how we get or we

1:16:51 calculate one of those scores. So, this score -0.3380 just of the first trajectory is an amalgamation of multiple scores. So, we’ve got the log score. We’ve got the spherical score. It’s a combination of those. So, uh the log score is trying to answer the question, “Did you put probability where the teacher put it?” Spherical is that does the whole answer

1:17:21 point the same way as the teachers? So, overall there has to be a trajectory towards approaching the right answer, but there also needs to be an approach where if a teacher model is scoring a particular option high, the student model should score it equally high. So, it’s mimicking the distribution on the one hand, being directionally consistent on the other. That’s the intuitive part. So, we’ve had uh the way of implementing it is that we have the teacher probabilities here 0.7, 0.2, 0.1. We have the student

1:17:53 probabilities here 0.52, 0.31, 0.16. I take the log of this, which comes to this. And then I weigh these logarithms by the teacher probabilities. So, the more probability it has for a particular option, the more say it will have towards the overall sum. Now, once I have this sum, uh I’m going to put it as a particular term, and then I’m going to sum it up with the spherical score. So, how does spherical score work? Uh it’s simply again

1:18:25 probabilities weighted by the probabilities of the teacher. So, we’re not only taking the log of the probabilities weighted by the teachers’ probabilities, We’re also taking literally the probabilities weighted by the teachers’ probabilities. So, we’re taking 70% of the first probability, the student probability, and so on and so forth. We’re normalizing all of that. And then, the final score comes out to be this, which we are

1:18:56 injecting here. So, these two put together constructs the reward for a particular trajectory. Now, uh this is where the nuances come into the picture. Um 99% of this is very clear. So, there’s a teacher uh model that spits out the probabilities. There’s a student model that’s trying to adhere to it and trying to mimic it. That’s the log score and the spherical score.

1:19:27 But, there’s also one last push only for one particular interface. So, remember uh Jev has the score, the null, the choice interfaces. Just for the score ones, another implementation that we have is something called RPS or ranking probability score. So, what we are trying to answer here is that score will have levels or it’s ordinal in nature. So, let’s say I have a question where

1:19:59 the answers can be um frustrated or very angry. Now, that’s a spectrum. It’s going from almost You can visualize it as going from left to right. So, let’s say the truth here says that the customer is very angry. So, it’s zero for calm, zero for frustrated, and one for very angry. Now, if I say frustrated versus if I say calm, you will notice that saying I’m frustrated is slightly better.

1:20:33 It’s I mean, I would say that the model is slightly better calibrated than another model that says calm. Right? So, and this only applies to ordinal predictions. So, for this if I uh if I actually predict this .1 .8 .1 then the log score or versus .8 .1 .1. So, if I just reverse this uh log score will give me the exact same

1:21:04 score. But, you see the RPS will differ. The uh probability score the running probability score uh will differ. So, it’s the squared gap between the running totals averaged over a minus one of them. So at every step I am going to uh take this RPS only for the choice questions and feed that back into my loss function in some way.

1:21:37 Yeah? So, the loss function is finally equal to the log score 75% of the spherical score and you can basically weight fully if you want to minus the RPS. So, this is the grade and in order to construct the final loss function um I’ve all I’ve actually seen things like uh cross entropy plus this RL loss. So we’ll go with full cross entropy plus the RL loss

1:22:09 and by RL loss I mean it will contain the spherical plus uh the log score. And if it is a score style it will also have the RPS embedded in it. So, that is the the core proposition of how RL CD works. So, there’s a full-on cross-entropy, and then there’s a full-on RL component.

1:22:41 And for that, the trajectories are generated uh with the help of GRPOs by adding random jitters, and it tries to come up with two loss functions combined to come up with the final RL loss, and then finally, uh you back-propagate that over whatever you’re training. You can choose to train over a simple linear head, or you can choose to train over uh a Laura, or you can uh do full-scale uh backbone fine-tuning as well. The choice is up to you, it’s up to your appetite, it’s up to the accuracy metric

1:23:13 that you want for your use case, and so on. It really depends. Quick note on the implementations and some of their comparisons, advantages, and disadvantages, and so on. Uh so, we’ve implemented both paradigms for Jev, uh the encoder-based and the decoder-based. So, uh some of the dimensions along which you should evaluate whether to uh take the encoder approach or the decoder approach are as follows. So, if the state is to be kept in between questions, the encoder has to reread it every time, but the decoder

1:23:45 can read it once and cache it. So, if you have a lot of questions over a finite set of context, I think decoder is the right approach. Now, cost of questions pile up, this grows with every question. Um this remains flat, only the questions are new. Uh if you want more context length, if you have longer documents to process, then decoder is the right approach. State read with the question in view. Uh again, since this is uh bidirectional implementation that encoders do, it will

1:24:17 always have a slight edge over uh causal mask or causal attention. Options are several tokens. Now, since the marker sits in front, it’s very easy to have uh mask tokens injected before those options and have them scale well. Uh again, with decoder, it will be slightly disadvantageous compared to encoders. Several labels at once, we can easily replace the head with a sigmoid. We are

1:24:47 only stuck with softmax, so we’ll really have to um do a little bit of model engineering, tweak the architecture a little bit in order to make this happen. Um >> [clears throat] >> now, typically lower uh number of parameters in encoders do a better job compared to their counterparts. So, if I have a 400 million parameter encoder model, uh I would need a more capable decoder model in order to do the same job. So, that’s the core aspect. So, if you’re really crunched for

1:25:18 uh memory or obsessed with speed, uh you really have to take into consideration all these dimensions along with your domain and problem statement in order to make this work. Uh and then finally, what you have to write is uh one sequence for question versus mask positions and cash. This is the engineering tradeoffs that you’ll have to uh take care of. Yeah. So, I think that summarizes all uh what we have tried to bring together here. Hope you enjoyed and really have fun learning this. Thanks a

1:25:49 lot for your time.