Can RL Make a 4B Model Compete with 27B?

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Can RL Make a 4B Model Compete with 27B? · Channel: Prompt Engineering · 16:06 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI models, Local and open-source models

Summary

The video tests whether reinforcement learning with open-source environments can improve a 4-billion-parameter model enough to approach a 27-billion-parameter model. Using GRPO and a stricter answer checker, the creator found that training on tasks the small model sometimes solved produced clear gains and closed more than half the performance gap. The experiments suggest that RL mainly sharpens abilities the model already has, and progress stalls when tasks provide no useful learning signal. Distillation changed the model’s problem-solving behavior, and adding RL afterward produced the strongest small-model results, though they still did not match the larger model.

Key points

An RL environment gives a model a task, a sandbox and tools, then rewards or penalizes its answer using a checker.
The model learns what the checker rewards, so a trustworthy and appropriately strict checker is essential.
With GRPO, the model generates multiple attempts per task; tasks where all attempts succeed or all fail provide no learning signal.
Training on tasks the 4B model sometimes solved improved its score and closed more than half the gap to the 27B model.
The trained model more often completed tasks and submitted the requested answer, although it also lost some tasks it previously solved.
Further RL training did not improve results when the selected tasks offered little new signal.
Fine-tuning on successful 27B-model transcripts changed the smaller model’s problem-solving behavior, and subsequent RL gave the best small-model results in the experiments.
The experiment was limited to one model, and the final small model still did not match the 27B model.

Chapters

Start	Chapter	Summary
00:00	The question and what RL environments do	The creator introduces the goal of improving a 4B model and explains how tasks, tools, sandboxes and answer checkers make up an RL environment.
03:39	Training method and experimental setup	The video outlines pre-training, fine-tuning and GRPO, then describes the hardware, LoRA training and held-out evaluation.
06:15	Sponsor: Firecrawl	A sponsored segment shows how Firecrawl can provide live structured web data for creating checkable tasks based on newer information.
07:47	Selecting useful tasks and measuring RL gains	The creator trains on tasks the model sometimes solves, reports improved scores and illustrates changes in how it completes data-analysis tasks.
11:23	RL’s limits, distillation and final results	Further RL alone yields little improvement, while distillation changes the model’s approach and a final RL stage produces the best small-model results, with limitations noted.

Quotes

A 4-billion model clearly got better and closed more than half the gap to a model about six times its size. — 01:02

the model learns whatever the checker rewards, not what you meant. — 02:35

The model only learns from task it sometimes gets right. — 04:41

RL sharpens what the model can already sometimes do. — 15:02

Concepts and entities

RL environments, GRPO, LoRA, reward hacking, distillation, Firecrawl, Opus 5.5, Claude

Full transcript (original language)

00:00 Can you make a 4-billion model comparable to a 27-billion model using RL? That’s exactly what I’m going to be showing you in this video. If you look at recent model releases, there is one thing that keeps coming up again and again, and that is RL environments. Anthropic, for example, has reportedly discussed spending more than a billion dollars on them in a single year. And it seems like having great RL environments can push the envelope up when it comes to model performance.

00:31 Now, the good news is we have started seeing some really incredible RL environments in open source. Xiaomi open source thousands of them with Mimo with 2.6. And now, even Hugging Face team has released RL environments that are specifically built for small models. So, for this video, I wanted to explore one simple question. Can you use these open environments on your own hardware to make a small model better? To find out, I ran a series of experiments on two DGX boxes that I

01:02 have. And I’ll walk you through all of them. Now, the short answer is yes, it works. A 4-billion model clearly got better and closed more than half the gap to a model about six times its size. But what I noticed was that at some point it stopped improving. And the reason why is probably the most useful part of this video. Now, before showing you my setup and my experiments, let’s talk about what exactly an RL environment is. So, a good mental model is a practice

01:33 exam with an answer key. The model gets a task, it tries to solve it, and something checks the answer. Now, to make this concrete, let’s look at one that I used, which is called small data environments. Every task is a real data set from Kaggle, which has a question about that data, and then an answer that a program can check. So, for example, here is a spreadsheet of video game sales, and the question is which particular publisher

02:03 has the highest total sales. For this a specific one, the answer is Nintendo. Now, and on each task, the environment gives the model three different things. First, a sandbox, which is basically a locked-down computer where the model cannot break anything. Second, is a tool. In this specific case, it’s a terminal. So, the model can look at the files and run Python. And the third is a checker. The model writes its final answer to a file, and the checker can paste it with the answer key.

02:35 So, if it’s right, the model gets a reward of one. If it’s wrong, it gets a reward of zero. Now, that reward is the only feedback the model ever gets. But, the most important thing to keep in mind is that the model learns whatever the checker rewards, not what you meant. And this is very important. If the checker is rewarding the wrong thing, your model is going to end up learning the wrong thing from the data. So, the checker that comes with this specific data set is a bit lenient. For example,

03:08 if the right answer is 42, and the model writes 42, yes, that will still pass. Now, a model trained against a checker like that can learn to pad its answer. And this is what we call reward hacking. So, when I looked at the checker, I actually had to write a stricter checker. And every score in this video is based on this stricter checker, rather than the lenient checker that comes with the data set. Okay, now let’s talk about where does RL

03:39 fit into training a model. So, models are usually trained in stages. First, there is pre-training, where the model reads a huge amount of text and learns how how predict the next word. This is where most of the models acquire their real-world knowledge and language understanding. Then there is fine-tuning. Supervised fine-tuning is a good example of this where it learns by copying examples of good answers. And it also goes from simple next word predictor to question answer machine.

04:11 Now, RL comes after that. Instead of copying the model, practices what a good answer looks like. So, for example, it’s going to try a task. The checker tells it how did it do and it basically adjusts itself based on the reward that it’s getting. So, you don’t really have to write out the answer. You just need a way to check the answer. So, for this experiment, the RL method that we used is called GRPO. For every task, the model makes eight attempts.

04:41 The attempts that did better than the group’s average gets reinforced. So, the model is more likely to attempt those again. And the one that did worse gets basically pushed down. Now, here’s a very important part of this whole thing. If all the attempts are right or all of them are wrong, none of them beats the average. So, that task teaches the model nothing. And this is really important to remember. The model only learns from task it sometimes gets

05:12 right. Not the task that it always gets right or always gets wrong. We’re going to come back to this in a bit. Okay. Uh let’s talk about the actual setup and what exactly I learned through these experiments. So, I used two DGX uh Spikes. Both of them have 128 GB unified memory. The model that I tried to uh train was quant 3.5 4B and instead of retraining the whole model, I trained a small add-on called LoRA, uh which is

05:44 a lot cheaper to train. To measure progress, I held back an exam that model never was trained on. So, this is basically a validation set. Now, this is also included task from the data set that it has not seen before. So, before any training, the 4 billion model was getting about half right and half wrong. Now, in this case, we’re using a 27 billion model. Uh this is the same coin family, but this was the target model for us. And the goal was to get to the same

06:15 performance as this 27 billion model. Let’s talk about today’s sponsor, Firecrawl. Remember, only 87 of the exam tasks used data sets that the model never saw in training. So, how do you get more? You could ask a frontier model like Opus 5.5 to write them. But, it only knows what it saw before its training cut off, which means the same famous data sets every model has already memorized. This is where Firecrawl comes in.

06:45 A general web search tools only scrapes the web data and can be blocked by certain websites. Firecrawl gives you access to live structured data from 100 and 10 plus million sources without the need for parsing HTML files. It’s the context API for AI agents. Through its MCP server, Claude could search Firecrawl’s developer index, found the 2026 World Cup data on GitHub, and pulled all 104 matches,

07:17 including every goal with its minutes. From that, it wrote a new task, which team scores the most goals from the 80th minute on. In this case, it’s Argentina with 10. The answer is computed from the file, so Checker can grade it. And the tournament ended after the cut off, so no model has memorized it. If you need live structured data from web, Firecrawl is your answer. Link is going to be in the description. And there’s a free tier for you to get

07:47 started. Now, let’s talk about the rest of the video. Okay. So, how do we close the gap of the performance? Now, I’m going to describe some of my experiments. So, first I train on a random set of tasks. The model did learn something useful. It started finishing its task instead of running out of turns. But the score only moved by a little bit. And you could easily say that was luck. Now, one of the interesting reason was coming from GRPO that I described before. So, when I

08:18 looked at the training tasks, most of them were ones that the model always either got right or always got wrong. So, essentially most of the training time was spent on learning absolutely nothing. Now, this is where I actually changed the tasks that I was training the model on. So, I had the model try about a thousand tasks four times each. Then I kept only the ones it solved some of the times and threw away the ones it always either was able to solve them or

08:49 never was able to solve them. Now, the reason was these are basically the tasks at the edge of what the model could do. So, I trained on those for most of a day. And it actually worked. So, the score went up clearly and it closed more than half the gap to the 27 billion model. Now, whenever you see a jump like this, the first question should be is this just luck? So, I compared the trained model to the untrained one task by task. Now, there was a subset of tasks that

09:19 the newly trained model actually gets right, but then there are another set of tasks that it gets wrong. So, if the training did nothing, those two subsets should be about the same size. Now, in my case, the first subset was a lot bigger. And the odds of that by chance are really small. Okay. So, we were making progress, but I really wanted to understand what behavioral changes were coming from the training. So, let me show you some real runs. Basically, same tasks before and after

09:51 training. Okay. So, remember the video game question that I described in the beginning? The untrained model actually does the analysis right, and it even says Nintendo in its reply. But, in the answer file, it writes the sales number instead of the name. So, it gets a score of zero. Now, the trained model double-checks the top five publishers and writes Nintendo. Now, I actually had to look at it because I wanted to make sure it’s not just a one-off. So, on questions where

10:22 the answer is a name, it writes a bare number a lot less often compared to the untrained model. Now, here’s another one. The question is, which game has the most playing time? Now, the file has no header row, which means the column columns don’t really have names in the file. The untrained model hits one error after another. It basically repeats the same command word for word, and essentially runs out of turns without writing the answer. The trained model figures out

10:52 columns, finds Dota 2, then writes it down in a lot fewer steps. Now, it’s not perfect. It also lost some task it used to get right. So, in this one, it does the math right, but writes a fraction when the answer key wanted a percentage instead. Now, I also wanted to check for reward hacking because the original checker was pretty lenient. But, in this case, I didn’t really see any indication of reward hacking whatsoever, which was great. Okay. So, the main finding was

11:23 that the model already knew data analysis, right? So, the RL part didn’t really teach it that, but it actually taught it to finish the job and answer the question that was actually asked. Okay, so the next obvious step was to do more auto, right? Again, I screened a fresh batch of tasks in the same way and trained again, but surprisingly it did not improve uh the model any further. And that’s the kind of detail that we talked about. We are looking for

11:55 tasks where the model uh if you run it multiple times, sometimes it can get right, sometimes it can get it wrong. Otherwise, there’s no signal for it to learn at all. Now, here’s a simple experiment that you can run yourself to see that ceiling. Give the untrained model four tries per task instead of one and count how many it solves at least once. So, that’s more than it solves in uh one try, and the gap is roughly the room RL has to work with. Now, my guess is that

12:27 training on the edge tasks had already used up a big part of it. Okay, so if the model cannot discover new skills by practicing, the next option is to show them to it. So, I had the 27 billion solve a batch of training tasks. Then, I kept the transcripts where it got the answers right. And then, fine-tune the small model on those. Now, this is basically the whole concept of distillation. In this case, we have a student model that

12:57 is learning by watching a teacher model work through problems. Okay, now when I tried this, the score actually did not change much. It remained pretty much the same, but the interesting thing is that the model started working like 27B model. So, here’s the question that asks whether one way of re-scaling the data gives a lower error than another. The auto model tries one machine learning model after another. The distilled model runs one clean comparison and answers in

13:27 a lot fewer steps, just like the 27 billion model would do. But I noticed something really interesting. The distilled model started solving some tasks the RL model never was able to do. So, the next obvious thing was that these are exactly the tasks that potentially would need RL. So, I ran RL one more time on top of it. Okay, so most of the experiments or all of the experiments that I ran were pretty much the same in terms of the final scores, but on the data set

13:59 the model had never seen this last model is the best of the small models that I tried. And I would say this was the first one where the gains passes the luck check. Here is one of those tasks. It’s a database from the US Department of Agriculture. And the question is the most common apple variety. Now, the untrained model and the RL model both answers novel. That’s the most common variety in the file. But it’s an orange. The distilled model finds the right answer, but turns out of turns while

14:32 double-checking. And the final model filters down to apples first and write red delicious, which is the same answer as the 27 billion model, and that is the correct one. Okay, so can you use an open RL environments at home to make small model better? I would say yes. You need a checker you can actually trust. Then you need a set of tasks at the edge of what the model can do. And probably a

15:02 day or so of training to get a first feel of whether it’s working or not. But the most important takeaway from this is that RL sharpens what the model can already sometimes do. Once you use up that, in my opinion, the scores basically stops moving. Okay, so take all of this with a grain of salt because this is a very limited experiment. Essentially, I tested this on a single model. Now, I tested a lot of checkpoints along the way and the last model is probably

15:34 the best one that I got. Now, it’s still not as good as the 27 billion model. It gets there, but I think with more training data, more diverse auto techniques, I could potentially get to the 27 billion model. All right. This was a fun auto experiment. I learned a lot and wanted to share this with the community. Now, I’m going to be putting everything on GitHub including the checker, the trained model. So, if you want, you can run this on your own GPU and see how far you can push it. All the recipes are going to be on GitHub. Anyways, I hope you this was

16:06 helpful. Thanks for watching and as always, I’ll see you in the next one.