Build & Train a GLM-5.3-Flash Model From Scratch with Python

Author: freeCodeCamp.org · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Build & Train a GLM-5.3-Flash Model From Scratch with Python · Channel: freeCodeCamp.org · 44:18 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI models

Summary

This tutorial builds and trains a small, 25-million-parameter GLM-inspired model from scratch, using a CPU-scale setup to demonstrate pre-training, vision, and reinforcement learning. It explains key architecture choices—including character-level tokens, hybrid attention, mixture of experts, and tied embeddings—then shows how experiments can test training and reward-design decisions. The central research lesson is to focus on carefully chosen questions, environments, data, and sound measurement rather than model scale alone. The speaker also emphasizes using small experiments, changing one variable at a time, and checking results across multiple random seeds.

Key points

The tutorial uses a 25-million-parameter model and a small character-level vocabulary so experiments can run on modest hardware.
Pre-training teaches the model to predict the next token by imitating text; reinforcement learning rewards correct task completions.
The architecture discussion covers hybrid linear and sparse attention, mixture of experts, hyper connections, RMS normalization, and tied input/output embeddings.
A small vision encoder turns image patches into tokens, and the tutorial demonstrates digit recognition.
Pre-training experiments compare data quantity, diversity, interleaving, and curriculum-style ordering.
The reinforcement-learning setup generates answers, evaluates them with hidden tests, and updates the model based on rewards.
Reward design, sampling group size, and generation temperature are experimental variables; results should be checked across seeds.
The speaker argues that designing useful tasks, data, and environments is often more important than refining an already adequate algorithm.
Small toy experiments can help researchers understand what is happening before attempting larger, more complex studies.

Chapters

Start	Chapter	Summary
00:00	Course goals and the role of AI research	The speaker introduces a CPU-scale model-building course and argues that researchers need to understand algorithms and choose useful experiments.
04:36	Next-token prediction and character-level tokens	The tutorial explains next-token prediction and uses ASCII characters as tokens to keep the vocabulary small.
06:42	Embeddings and transformer output	Tokens become learned vector embeddings, which the transformer processes to produce probabilities for the next token.
08:48	Model architecture and residual pathways	The speaker introduces the model’s hybrid architecture and explains embeddings, hyper connections, normalization, and tied weights.
15:04	Position and sparse attention	The tutorial describes positional encoding and a sparse indexer that selects relevant tokens for full attention.
18:14	Linear attention and mixture of experts	Linear attention is presented as a fixed-size memory for recent context, alongside sparse attention and expert routing.
23:00	Vision encoder and image tokens	The speaker shows how image patches can be merged into tokens and used by a small model to recognize handwritten digits.
26:10	Pre-training and data experiments	The tutorial explains imitation-based next-token pre-training and compares experiments involving data diversity, interleaving, and curriculum.
30:18	Reinforcement learning and evaluation	The speaker contrasts reinforcement learning with pre-training and describes generating task solutions, evaluating them, and updating the model.
34:00	Reward design and experiment priorities	The tutorial discusses reward choices and argues that task environments and data often matter more than changing an adequate algorithm.
36:04	Group comparisons and updating model weights	The speaker explains comparing multiple answers using relative rewards and describes updating only selected model components in this example.
39:11	Results, statistical significance, and research practice	The closing section reviews task-specific results, cautions about bugs and statistical uncertainty, and recommends controlled experiments and consistent reporting.

Quotes

we just need small models with fast iteration. — 04:36

So, it’s better to interleave them — 28:44

setting up question for the environment, for the data, and what is it learning is more important. — 35:33

you want to change just one variable at a time. — 40:44

Consistency is more important than intensity. — 43:48

Concepts and entities

GLM-5.3-Flash, Transformer, atenção linear, atenção esparsa, Mixture of Experts, RMS norm, pré-treinamento, reinforcement learning, Codex, Claude Code, freeCodeCamp.org

Full transcript (original language)

00:00 Learn how to build and train a 25 million parameter GLM 5.3 flash model from scratch using just a standard CPU. This tutorial covers core pre-training and reinforcement learning techniques, teaching you how to design effective experiments and think like a modern AI researcher. >> Let’s build and train GLM 5.3 flash, which is ox alpha, which is the latest best viral model from Google AI. We will

00:30 do both pre-training and post-training as well as building everything from scratch. So, you will learn how to do everything GLM researchers, everything researchers at open AI Anthropic do, just on a simpler scale. And AI researcher job has dramatically changed in just a few months. So, this is the latest up-to-date course that you’re going to need if you want to become AI researcher. All of the code is in the GitHub. These are the experiments. We will reproduce everything. You see the charts. So, it’s

01:00 below the video. This is the latest job posting for Anthropic, and this job posting describes 95% of AI researcher roles. First, you will understand the architectures and the algorithms in this course. Then, I will show you how to do actual AI research with designing environments, reinforcement learning, asking questions. So, right now, Claude code, Codex AI is coding all of the experiments, and it can do them autonomously. Your job as AI researcher is just to have high-level

01:31 idea which experiments to do. So, even though Anthropic and OpenAI could actually be inventing new algorithms, right now, it’s transformers, it’s reinforcement learning. I think they are focusing 99% or more of their efforts in just designing data, environments, and then optimizing these algorithms that you’re going to understand in this video. And thinking for the future, AI researcher Your is mainly going to become a high-level ideas and not coding. AI models are going to do all of the coding and

02:01 experiments. So, but you still need to understand how the algorithms work. You don’t need to write the code manually, but still people, humans are ones who can invent better, faster concepts, faster algorithms, and create better designs for the system. And then AI can implement, experiment, try out, etc. This is why we will still go over the code and understand how it works, even though we don’t necessarily need to write everything from scratch and manually. But if you want to invent new algorithms

02:32 as AI researcher, you need to understand how the current algorithms work. So, even though the original model is insane, billions of parameters, our model is just going to have 25 million, and it can be trained on a CPU, GPU, MacBook, anything. I made it very simple. And this is what you want to do. You can still do so much research here. You can learn how to ask questions, how to set goals, topics, research directions from for the AI. Uh even if you have a lot of compute, there is nothing more you can do with it

03:03 than with just a CPU. I don’t think unless you want to build a model that other people use. But here we’re trying to learn. So, we want to learn how to ask questions mainly, and AI can help answer those questions very easily. And AI will become more and more capable of answering questions, and humans are going to be asking questions for now. Of course, we don’t know what’s going to happen in a year, super intelligence, whatever. Then humans will be useless. Then AI will do everything. So, for now, we are just focusing on learning how to ask questions, ask research questions,

03:34 setting goals, directions. Here are some example questions we will be asking and answering in this research. So, this is real stuff you can do. You can learn how to research on just CPU. So, let’s say you are training some reinforcement learning reasoning. So, should you reward imperfect score, partial score? Should you not? Well, we will try out. We will see which one works better. How strong should you punish invalid code? And then temperature or randomness of next word generation in reinforcement

04:05 learning. So, it’s exploration versus exploitation with what it already knows. So, these are like a bunch of examples and just asking these questions is your job. This is what distinguishes experienced good researcher versus novel not inexperienced. And AI can do all of these experiments by itself right now. Even Soul Opus and then Astra Fable, everybody can do these experiments. By the way, if you need help answering asking these questions, you want to join a real AI research lab, join our school

04:36 community where we simulate a real AI research lab. You will be either setting your own research goals or using ours, answering our questions and helping us figure out questions. We are simulating teaching people, coaching people how to do real AI research. Okay, so let’s start. We know that LLMs are going to predict next word or next token or next character in our case because we are going to build and you are also going to build a small model. As I said, we just need small models with fast iteration.

05:06 So, in this example, input could be return X plus and the model is going to predict one depending on the task, depending on the prompt. This is the next token. So, the real real LLMs use BP tokenizers. Byte pairing coded tokenizers. So, they would If you have some text like this, they would split that into subword tokens or word tokens. Usually, they have 150,000 or 300,000 of these subword tokens and they can then encode

05:39 any text. Any text is contained in uh with this vocab in in this vocabulary. But to make it simpler to understand, to train, we are going to make each ASCII character. So, each letter is going to be its own token. Here, if we have this function def uh int whatever, each of them is its own token. So, we just have 260 uh tokens in our vocabulary. So, we don’t need to do uh tokenization training. It’s simpler.

06:09 Also, big problem that would happen later. So, some of you may already know uh transformer architecture, LLM architecture, and you know that there is embedding layer, there is output head layer. And both of those weight matrices use token vocabulary as one of their dimensions. So, if we have massive tokens, massive vocabularies 250,000, then on a very small model, those matrices would be majority, 95% of parameters

06:42 would just go into the vocabulary conversion. And other 5% of parameters would go into actual layers and the model itself. So, if you did not understand what I said, it’s okay. Just for a small models, you also need small small vocabulary. 260 token vocabulary is good. So, this is how uh next token gets generated within the transformer. So, let’s say we have some input. So, it can be some number, some prompt, whatever.

07:12 And each of those tokens in the input, uh each of them have corresponding uh vector embedding that represents them. So, each of them gets replaced with its own vector embedding, and vector embedding is like a sequence of numbers that encodes the meaning of that token. So, uh token A has some numbers, token B has some numbers, plus has some, minus has some. And then the vector for plus would kind of explain or contain how addition is

07:44 done and what is the purpose, what is the functionality of token plus. So, every token in the prompt gets replaced with its corresponding vector that describes its meaning. These vectors are learned by the LLM when we train the LLM, these vectors are learned. So, LLM learns how to represent concept of a plus with some numbers. And then we have sequence of vectors, that’s our prompt, that goes into the

08:16 transformer into the layers. It gets processed through the transformer and then at the end we want to predict the next token. So, it’s going to it’s going to just transform whatever was processed, the entire prompt was processed, it’s going to transform whatever is at the end into probability distribution over entire vocabulary. So, so based on the prompt, you will get for each token in vocabulary how likely it is to be the next token.

08:48 That’s roughly how it works. So, that is the very basic idea of transformer. Now, GLM 5.3 flash has some new architectures, new parts that we didn’t learn about before on my channel. So, linear and sparse attention hybrid and then mixture of experts, we talked about this, it’s very famous. And then deep six, manifold constrained residual connections, hyper connections. And then this one is quite famous and old, it’s to use the

09:18 same matrix for input and output vocabulary into embedding conversion. So, input and output. These are the specifications of our model. So, original is huge, but our is tiny. These are the layers, a lot layers, a lot fewer experts. And the context is just 192 tokens, while a GLM has 1 million tokens. And you will see that in our experiments, before reinforcement

09:49 learning, it will not do any of the tasks successfully. And after reinforcement learning, it will do uh 16 out of 24 tasks successfully. Those tasks will be very simple, like write some Python function that multiplies input by two. But you will see the difference uh before and after RL training. And you will learn how to ask questions, how to design ideas and environments for the uh model to learn. You can see also during training how the model is

10:20 learning. During This is the graph that shows how it’s learning during the training. Let’s look at the code now. So, we are going to understand the algorithms, how they work, but nobody’s going to be coding this. So, uh but it’s up to humans to understand algorithms and then improve them. Tell the AI what to try, what to And then AI is going to just try many ideas based on your idea. But you need to know the direction. So, this is what We’re going to be understanding this part of code uh now. So, first you have uh embedding and then

10:51 passing input ID. Let’s say this is our token ID. [clears throat] This will just look into vocabulary and pick the vector that I mentioned earlier. And so, this could be a huge matrix if you have huge vocabulary. And this could be huge uh weight matrix. And if you have a small model, then this would be 95% of parameters. This is why for small model we use small vocabulary, bytes. So, then it’s just going to retrieve uh this token’s vector. And this is at

11:24 the beginning when it’s processing the prompt. So, then we have num So, we after we convert all of these vectors, we we’re going to have like this is the dimension of the vector and then this is the number of tokens. So, number of tokens, token dimension and batch is like how many independent conversations you have in this huge X vector X tensor. You see how it will just take this X vector and create array of these X

11:55 vectors. These will be the residual connections. So, later each of the residual connections, so these are manifold constrained hyper connections by DeepSeek and ByteDance and each of them is going to carry different information forward without that information being processed through the attention and so it’s going to be mixed up, it’s going to be like carried through forward. So, it’s like some highways where you pass before different types of information. So, I think I will not go into too much

12:27 detail here. If you are interested here, I made some videos and you can read paper because I feel like it does get boring if and I cannot help you too much around this. You need to practice this yourself, read this, understand it, try this out. If I just explain, you will not be able to follow maybe. So, let’s just go through these things. Those hyper connections are going to get mixed into each other depending on some weights and ways of mixing. So, they are sharing information between each other between the attention feed forward

12:58 networks and all of this. And at the end they’re going to get merged into a single hidden dimension vector. This way with these hyper connections, you don’t just have one residual connection. You allow the transformer to form and to pass many different types of data through it while processing. And then that hidden state gets transformed into probability distribution over entire vocabulary with some weight matrix. So, distribution of which token is

13:30 likely to be next that fits in into that context. And this is the other part that can be absolutely huge compared to the rest of the model if the vocabulary is big and the model is small. So, this weight matrix that transforms that’s hidden state times um vocabulary can be big. And if you remember at the beginning of the Transformer, we also had vocabulary times hidden state or hidden state is actually vector embedding. So, you can actually use

14:01 uh this same weight matrix for both of them. So, the vocabulary, the picking the uh vector for each token and at the end here generating probability distribution you can use the exact same matrix and it works well depending on the architecture um they use it here. And then you save a lot of parameters that you can use at different places. Another part is RMS norm. So, this is just going to uh normalize vectors. So,

14:33 you see how we have like a very different number here than these. And neural networks and the way neural networks back propagate and do math calculations, uh this can mess with the calculations, make the weights explode, make them very tiny, or uh this can carry all of the weight and then make these irrelevant because this is so much bigger. So, usually you want to make the numbers around 1 -1 0 uh and uh similar in scale.

15:04 And then these numbers will not go crazy when you multiply so many of them in a row. So, this is positional encoding. So, how does Transformer know which token comes after which one? Because in attention in attention mechanism, uh you have tokens multiplied by other tokens, but you don’t know the order of tokens in the prompt. So, each of the token vectors is rotated. For example, if you have some vector, you take pairs

15:36 of dimensions and rotate each pair. And depending on where uh the number of like where in the sequence that pair is and where the token is in the sequence of tokens, you’re going to have a bit different rotations. So, Transformer will learn based on how rotated these vectors are uh their position. So, in GLM, they are using NOPE. That’s a version of Rope in main model and then

16:07 Rope in the sparse indexer. This sparse indexer uh is checking in the attention which part we should it should attend to, which part it should not attend to because the way attention works here is it’s not going to attend to everything uh entire attention entire context window. So, this token will just attend to most relevant tokens. So, this indexer is going to search for the most relevant tokens for this token to attend to. And indexer is also attention mechanism, but a lot

16:39 lighter, a lot cheaper, a lot faster attention mechanism. And it just looks for relevance and it doesn’t need to do so much computation that uh regular attention does. So, it picks most relevant tokens and then only those tokens are uh computed with the full regular attention. So, for each token in the attention mechanism, it’s going to pick uh n most relevant tokens. It can be 2,000 most relevant tokens or 1,000 most relevant tokens and then do attention

17:10 with that. And then here, the attention is just scaling based on this small indexer. So, this small indexer So, instead of scaling like this, it’s going to scale very a lot slower because because the attention size is always fixed, it doesn’t scale, but the indexer scales with the sequence length. Sequence length. So, I’m talking about how compute scales with a sequence length, and it’s going to scale a lot slower with sequence length, a lot less compute because

17:43 this sparse indexer is a lot lighter type of attention and does a lot less computation. Just needs to find relevant tokens. So, as you can see So, the linear attention is different from this standard quadratic attention. It keeps So, not to confuse you, let me first explain that their attention mechanism does not contain positional encodings. And only the sparse indexer contains

18:14 positional encodings. And so So, nope means no positional encodings. So, we apply rope in attention and indexer just to make it simpler for our model. And linear attention is just KDA linear attention. So, this is state space model. I’ll explain this. So, we have linear linear linear sparse. So, linears are cheap. They always keep a constant vector. So, imagine that

18:45 you have tokens, you have this like state like a matrix that that’s like a memory of all of the tokens. And all of the tokens that as they come, they get fed into this state into this memory, and it’s always same size. It’s like just adding it in there. And so, linears are going to remember recent tokens. They’re not going to remember past tokens. Because the further away token is, the more it’s been modified, it’s been messed up by the newer tokens. So, this is for the recent tokens.

19:16 And these are so fast, a lot faster than attention. And attention is for uh long-range any tokens anywhere in the context window. So, by my experience, this trains a lot faster and inferences a lot faster as opposed to attention. That’s why this is optimized for inference. So, linear layers carry compressed history cheaply, memory. Sparse layers periodically recover exact distant details.

19:48 Here is the code for the layers and attention. So, let me show you it’s very simple. So, you’re just going to take every fourth layer and set it as sparse attention. And this is linear attention. So, you have some tokens and they all get into this matrix memory matrix. They call this running key-value state. So, as context grows, sequence length grows, this state stays fixed. So, this is a way to remember uh stuff without

20:19 scaling quadratically. The trade-off is you lose some information because you are compressing it. So, sparse attention usually in sparse attention you look at uh fixed window of last n tokens, last 2,000 tokens. And then you can uh look at regular uh spaces or you can search for relevant tokens uh behind. And then we have mixture of experts. For example, uh in GLM you uh you have 288

20:52 uh chosen experts. And it’s going to choose each token will pass through eight out of 288. And there is one shared expert. Every token passes through the shared expert and that expert will uh learn some common information for every token, so other experts don’t need to waste capacity on that. So, we may have uh eight routed and two uh two out of eight chosen, and then one shared in our small model. I also had I

21:25 can have a model with just two experts and one routed. I mean, one chosen. Here I show you the model configuration for our training model. We already mentioned uh this, so eight experts, 12 layers, etc. So, there are these four hyper connections. Then, you have uh inside of attention. So, between attention and then MoE So, they kind of merge into attention, then they expand back, and then merge back into MoE, and then so

21:56 So, here they are connected like residuals. And here they’re also connected like residuals, so they pass MoE, and they go through MoE as well. So, attention moves information between token positions. MoE applies specialist transformations to each position. Residual attentions preserve the previous representation while adding both updates. And multiple connections preserve different type of information. So, input and output share one matrix,

22:27 so the output weights and embedding weights are the same matrix, as I mentioned earlier. So, read uh byte tokens and predict next byte the same weight matrix. One matrix learns both token embeddings and output logits. Or I should say the weight matrix that transforms them into those. So, weight tying, this is how it’s called, remove second large matrix and makes reading and predicting bytes use the same learn geometry. So, maybe there

23:00 is some benefit there as well. I don’t know if there is research, probably there is. Maybe you also can do some. So, there is also vision in GLM flash and it’s So, there is image, it’s going to convert that image into some patch embeddings, convert those into tokens, and then mix them into the main model. For images, it’s going to get divided into patches. Each patch is a token, so it has this 2D positional rotary embeddings.

23:35 Similar to rope, it’s just 2D. So, let’s say you have image number eight, it’s 32 by 32 RGB image. And it’s going to get divided into 8 by 8 patches in this case. So, there are two vision blocks in our small version that we coded, that all of this code is in the GitHub, by the way. All of the code right here in the GitHub, so you can check it. These are the experiments.

24:05 So, each of those patches, four patches are going to become one vector, one token embedding vector. So, 2 by 2 merge. And so, we will get 4 by 4 at the end, which is 16 tokens from this 32 by 32 image. So, if you have some other image that’s not 32 by 32, it must be resized into this shape and padded there or cropped. So, this is just for our encoder. GPT

24:36 has any image encoder, you know, these other models, and we can also make any. I just made this one very quickly. So, image tokens are going to have some mark, like image starts here, and then all of the 16 image tokens, and then image ends here. So, this is how they get added into the context window. There are these marks. And you may have a uh text around. So, this beginning of sequence, by the way, I didn’t mention this is special token

25:08 uh in the vocabulary. It just shows this is beginning of sequence of a conversation. And then you can append the text to all of these tokens. And then the next here, the next one would be, for example, number seven, predicted token. If this image is of uh image is a digit seven. So, we trained a small transformer to look at the image of a digit and tell us which digit it is. So, if you are interested more about the

25:38 nitty-gritty details, you just tell your Codex or Claude Code to clone this repository. You can ask it about the code, about everything, how everything works. You can also tell it to draw these graphs, like I did here. So, these are examples. Now, of course, it’s very visible to us. It’s very easy for us to see. Anyways, our vision encoder, our vision transformer was successfully recognizing digits using this method that I explained. If you want to do some research, you can make these a lot harder and then try to

26:10 see at which point it’s not able to recognize them. So, let’s look at the pre-training. So, we have some prompt that is a Python function. And then we have this completion, which is model generated. Uh because the task is return x + 1. Now, uh in pre-training, it’s not uh thinking of this answer. It’s just repeating what the data is. So, it’s just repeating uh these. That’s how it’s learning. It already has all of these.

26:41 So, it’s similar to imitation learning. It’s imitating. Now, in pre-training, it’s imitating to write. It’s learning what all of these tokens, all of these characters mean, so it can generate them. And then in reinforcement learning, it’s going to learn to generate, to think about which answers are actually correct. So, it’s learning in post-training, it’s learning to be intelligent more. So, you have a red it predicts you if you have a return predicts the space. You have so many training examples just

27:13 from one prompt. This is the classic pre-training pipeline probabilities cross entropy assigned weights and then optimizer is going to update all of the parameters based on result next token correctness. Uh so, AdamW most of people are using me on, but I just did it for simplicity, so we don’t use both, we just use AdamW. You see how gradients go from wrong probability distribution to more correct one.

27:43 They are literally moving the choice. It’s going to change millions of weights, so correct byte becomes a bit more likely each time. You see how in the beginning, before pre-training, it was just generating some random uh characters if this is this is the prompt. But, later it learned to generate this. Now, let’s see some research questions and experiments I did. So, this is how they do research. This is very standard example, so you

28:13 see if you have less uh data and you repeat the data, it’s going to learn faster in the beginning, but later more data will just overtake. So, diversity is not useful if you do like just a few steps, but then uh at 200 steps, it becomes more and more useful. Okay, now let’s say you have uh three different types of data, type A, type B, and type C. Is it better to uh first train on type A, then type B,

28:44 then type C, or interleave them? So, uh you will see that if you train it like this, then as it starts to learn C, it’s going to start forgetting A and B. So, it’s better to interleave them and you can show this with an experiment as well. So, you can do this experiment if you want. So, interleaving data will improve generation as well as well as prevent catastrophic forgetting of previously learned data. So, homogeneous

29:16 blocks likely increased recency bias or forgetting, but you need to know what your experiment is measuring. Here, we’re just measuring performance. We’re not I’m not measuring if it’s increasing recency bias or forgetting, so I cannot make those claims. I can just suggest that might be the case. How about this? What if we first teach it repeated patterns, simple patterns, and then after teach it more diverse data? So, maybe in the beginning, it’s simpler things to learn and then later more complex. So, we are comparing that

29:47 to only simple things and only complex things. And so, we already know that simple things, they underperform only simple only same data, it overfits. And here, curriculum did not make substantial progress as opposed to just training for the the complex data from the start. Maybe this complex data was also too simple, so I need to think carefully about my experiment setup. Also, this difference may not be statistically significant, depending on how many times I measured, how I

30:18 measured. I need to think about all of these things, and AI can help you understand and think about all of these things and setup of your experiment. Now, let’s take a look at the reinforcement learning. So, here reinforcement learning is different than pre-training. Pre-training is just learning to repeat text. And it’s learning what each token means. In reinforcement learning, it’s trying to generate answers, and then we reward correct answers. So in some way I don’t know how but it’s learning to think somehow and if you can figure out how

30:49 you’ll be the most famous AI researcher. But in reinforcement learning it’s learning to use its thinking throughout transformer to figure out correct answer. Right now reinforcement learning traces single rollout single task completion is taking maybe even days in frontier labs like open AI and Anthropic. If you want the model if you want to teach the model to work for days you’re going to during the training you’re going to let it try to complete tasks that take days. So

31:23 this is just by the way if you are if you want to help these labs you can also work on inference. Faster inference means you are generating data faster you are you can train faster. So generate code run tests score which is correct which is incorrect now. Different ways of scoring you can score only correct or partial results for example and then update the model weights. You will not show tests to the model. You will just evaluate its result because if you show test it can

31:54 reward hack which means like maybe reading the result instead of learning how to solve them. Learning how to hack into hugging face or different systems what happened and uh in our post training of a small example we went from 4% correct answers to 45% correct answers after the training. So uh usually you can just tell Fable Fable and Opus and Soul to do this post training for you. Uh this is what you need to learn what Opus and Fable they don’t know is

32:26 get the experiment direction get the experiment ideas. So they can do reinforcement learning trade training simply because it’s been done so many times on the internet they know how to do it. But your judgment, right now your judgment for experiments is what you need to practice. What kind of ideas? And then it can do the experiments. So before reinforcement learning zero out of 24, after reinforcement learning 16 out of 24 uh questions. Pass at one means you just give it one try

32:57 and then judge if it’s correct or incorrect when you are testing it. If you have pass it eight, it means you are giving it eight tries and if any one of eight is correct, so you would take it as correctly answered. I mean, you just give it same question, let it try to answer multiple times. But here we just let it answer once and then see if it’s correct or incorrect. Of course, you will always need to see how you are measuring your results. For example, in this case uh we are testing it on similar tasks

33:28 that it learned uh during training. It’s not same, but similar. Because then it’s going to learn templates, it’s going to learn how to do these tasks and then it’s just going to learn to apply to different numbers, maybe different operations. Now, the more diversity you want, the more uh generality you want, you need to scale the model. More parameters, more compute, more data. And then it’s going to become more general, more more diverse. But don’t worry about that. You can always do research. I think you can do good research even on a CPU. I feel like

34:00 um if you want to get a job at OpenAI and Anthropic, you don’t need GPUs. You want to learn You want to teach yourself how to construct environments for reinforcement learning, how to think about them. Check if model is learning in those environments, that’s important. It’s not important that you uh scale unless you are working explicitly on scaling or pre-training. If the task is return 2 * X before reinforcement learning, it would return X * X and after reinforcement

34:31 learning, it would return x times two. So here it generated correctly, it learned. So what reinforcement learning did is increased probability that the correct answer will generate based on all of the parameters. There are many ways in which you can structure rewards. For example, you can say plus one for all tests passed, uh zero for valid format but wrong number. And maybe minus one for invalid Python syntax. So this is just one way. You can

35:02 just give zero for invalid uh or wrong and then one all tests. You can also give one if it writes a correct format as well. If it writes a correct Python, you can reward. So there are many different ways. And this is experimentation, empirical evidence, um in but I also think like it’s a lot more data is a lot more important than structure of these algorithms. So you will find a good enough algorithm and then you will uh

35:33 have a diminishing returns on thinking about the algorithm as opposed to thinking about data. You will improve the model a lot more if you are thinking about what am I teaching it? What is the environment? What is it trying to do? Can it do it? Can it learn it? So setting up question for the environment, for the data, and what is it learning is more important. Is the most important thing once you have good enough algorithm. This is why Deep Seek is no longer uh putting substantial effort into GRPO. GRPO, the reinforcement learning

36:04 algorithm, is good enough. If they put more effort, it will not yield as good returns as if they put more effort into data training, post training, environments. The verifier is going to run hidden tests. Now you don’t show these tests to the model. Now you have like GPT, Fable, Astra, they are going to they are possibly going to hack and see your uh um verifiers. So, this is something that’s happening right now. So, there are different ways you can do this. So, for example, uh in GRPO and

36:36 other algorithms, you can uh give the same question and let model generate 16 answers. And then you can reward each answer based on if it’s correct, incorrect, wrong Python, etc. And then we want to calculate how much each of the answers is better or worse than other answers, than the average. So, they call this advantage. So, reward is just your 1 0 or -0.1. That’s your reward.

37:07 Minus mean of other rewards. So, it’s just going to calculate is this better than average and how much is it better or worse than average. So, better ones are going to become more likely and worse ones will become less likely. This is good because you don’t need separate critic, like separate model to uh review the results and score the results. You don’t need humans to do it. You can just let it generate multiple times and compare them to each other.

37:37 So, I’m not going to update the entire model with these reinforcement learning signals. I’m just going to update uh last block final normalization layer and the output head. Those are tied weights between input and output as I mentioned earlier. But just the end because this way, first of all, we are not changing previous layers that have some other knowledge and then they might be they might experience forgetting of other knowledge because now we are just changing on this new data. We are not training it on the old data.

38:09 We So, it it might change stuff too much. And this is also faster, up- updating less weights. So, you can also use Laura. You can use different techniques. You can update the whole model. So, there are different techniques here. Um you are going to look into this if you want. You can also do research and experiments and try different ways of updating a few blocks and try to understand why. So, so try to uh when you do experiments, uh use toy examples. And this is what MIT says. MIT says you need to use start

38:41 from small toy examples so you can understand what’s going on. It’s better than doing huge experiments, complex that you don’t know what’s going on. R L O O is the algorithm I’m using. So, algorithms, as I said, doesn’t matter too much. Data is what matters. What is it going to learn is based on which data we give it. So, because our model was small, we are able to improve it only around similar tasks that we train it on. But on the other hand, if you have big

39:11 models like OpenAI, they are also going to train it on other different tasks. So, you could say that they might also only improve it on tasks they trained on, but they train on everything. And this is a joke uh that people complain that LLMs only learn what it’s is in training data. And then the other guy says, “Then we’ll put everything into training data.” So, we had these operations, Python operations. So, increment, double, even. Um these are just some Python tasks and

39:42 it improved on these. And now, interestingly, it didn’t improve on this one. I don’t know why. I need to maybe check it. So, so here, maybe it’s a bug in my code or bug that AI generated or bug in my experiment or something. Or if it’s not a bug, then it’s even more interesting, but it’s probably a bug. And then ones that it didn’t train on, it uh didn’t improve. It may even have regressed. Now, depending on how we calculated the statistical significance, you need to always be careful.

40:13 Did it actually regress? Do you have enough samples? Did you calculate it properly, they do sample maybe multiple times to see the average around different uh percentages. So, two previously strong families regressed because only a narrow parameter surface was optimized for the practice reward distribution. But, it did improve in the things we trained it based on our environment. And this means that you can teach it whatever you want to teach it and know

40:44 how to construct environments. So, when experimenting, you want to change just one variable at a time. So, let’s say you have all of these different variables like reward, uh how you’re rewarding, the group size, how many uh answers you are generating at once, for example, eight per prompt, temperature is randomness or penalty, negative rewards. So, you want to adjust if you’re doing research, you want to do one uh experiment, one variable per time.

41:14 Here, I did a bunch of different experiments. You see, if I set, for example, group size four, it performs very well, better than group size 16. Then, temperature 0.2 is uh more deterministic, less random, performs better than more random temperature. And then, uh here in these experiments about reward binary like uh 0 1 or other partial rewards or no invalid penalty, hard invalid penalty, they all perform same on my data set. Now,

41:44 you don’t want to just have five, you want to have like 500 or 5,000 of these to get uh better insights. But, check out this horror. You see, group size four won over group size eight on this seed, but lost on this seed and lost on this seed. Okay, so this is This means that, first of all, this doesn’t have much influence based on my setup, my training data. Secondly, we Yeah, we need to train more to see actual statistical significance.

42:16 Um and then also it could be that it’s not so important. But, if you are training huge models, then even small differences could mean a lot of money saved. Your model is training a bit faster, which is going to save a lot of money. So, this is why when people are reviewing your papers, they will tell you to test on three seeds. And all three seeds should have a clear conclusion. Like, all three seeds it should say that group four is better than group eight.

42:46 So, none of my experiments produced statistically significant improvements. Maybe my experiments are very small. Maybe as you probably as you train bigger models, you’re going to have uh clearer and clearer exact setup of parameters that’s going to be best. And here we can review each part. So, pre-training is going to teach it to write Python. Revision is going to teach it to add recognize images into the transformer into the language model. And

43:16 reinforcement learning will in our case improve the increment and double operations. So, I encourage you to just go talk to Codex or Cloud Code. You can tell it, “Which kind of experiments can I do here?” And don’t be so concerned about scaling. I just want you to learn how to do some experiments, how to learn scientific thinking, how to measure properly, how to conclude properly. So, you can post your experiments on social media as well. If you want to become AI researcher, I recommend you just keep posting your experiments every day or

43:48 every week on social media. Consistency is more important than intensity. So, if you post five days in a row, but then you stop, it’s not good. If you post once a week forever, it’s a lot better. And it’s better to just do experiments and train a model for a few seconds. You don’t want to wait a few minutes, a few hours because it’s wasted time. You need to be thinking how to need to be practicing how to design experiments, environments, questions. You don’t need to wait for

44:18 the model because you cannot train the model that’s going to outcompete these companies that people are going to use.