Xiaomi Just Open-Sourced Its RL Training Secrets — I Used Them to Train My Own AI

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Xiaomi Just Open-Sourced Its RL Training Secrets — I Used Them to Train My Own AI · Channel: Prompt Engineer 48 · 16:15 · Watch on YouTube

Topics: AI content generation, AI models, Local and open-source models

Summary

The video demonstrates training a music-writing model with Xiaomi’s open-source reinforcement-learning environment, grader, and training tools. Using one rented GPU for about 1 hour and 40 minutes at a cost of roughly $12, the creator increased playable outputs on 40 unseen prompts from 1 out of 160 to 19 out of 160. A key obstacle was that blank lines in ABC notation caused otherwise usable music to receive a zero score, so training used partial credit for those cases while evaluation kept the original grader unchanged. The creator shares the training setup, results, and a merged model and adapter for others to try.

Key points

Xiaomi released around 7,000 reinforcement-learning environments, including a music environment with prompts, verifiers, and training recipes.
The music task uses ABC notation, which is converted to MIDI and audio for evaluation and listening.
GPRO compares multiple answers to the same prompt against their group’s average reward, encouraging better-than-average answers.
The music grader evaluates 18 measures across six areas and assigns zero to malformed ABC, including pieces with certain blank lines.
Removing blank lines from failed outputs raised the valid rate from 1% to 16%, revealing that formatting—not necessarily the music itself—was a major problem.
During training only, pieces that failed solely because of blank lines received 30% credit; the final evaluation used the unchanged grader on 40 unseen prompts.
After 30 training steps, playable music rose from 1 out of 160 outputs (0.6%) to 19 out of 160 (11.9%), while average answer length fell by about 40%.
The creator produced a roughly 700 MB LoRA adapter, merged model, and GGUF version, and describes how to run the model locally.

Chapters

Start	Chapter	Summary
00:01	Training a model to write music	The creator introduces the before-and-after music results and explains the video’s low-cost, end-to-end training demonstration.
03:13	How GPRO and the music grader work	The video explains group-relative rewards, the length penalty, ABC notation, and how the grader evaluates generated music.
06:01	Testing base models and finding the blank-line issue	Initial model tests reveal that blank lines in ABC notation caused many otherwise promising outputs to fail evaluation.
08:10	Preparing the data, rewards, and training code	The creator walks through the held-out test prompts, Xiaomi’s grader, the training-only partial credit, and the GPRO configuration.
11:22	Running training and evaluating the results	After resolving configuration and memory issues, the creator trains for 30 steps and reports improved results on unseen prompts.
13:01	Results, limitations, and listening test	The creator discusses the increase in playable outputs, notes that the model is not yet a finished composer, and presents a blind listening test.
15:11	Trying the model and next steps	The video demonstrates generating and playing a piece with the released model, then suggests ways to extend the experiment.

Quotes

we went from less than 1% playable music to almost 12% — 00:33

RL can only improve what the model can already do sometimes. — 04:53

74 out of 80 failed pieces had this. — 07:39

After training, 19 out of 160, 11.9%. — 13:01

finding the blank line problem was the real key. — 15:43

Concepts and entities

reinforcement learning, GRPO, ABC notation, MiMo-V2.6, MiMo 2.6 Distill 9B, Qwen3 8B, Xiaomi, Hugging Face, RunPod, TRL, GGUF

Full transcript (original language)

00:01 All right. In this video, we will see how I trained a model to write music. And you can see that on the left, this is the base model. It cannot write even one playable song. And on the right, this is the same model after reinforcement learning RL. And you can hear that [music] it actually is playing some music. Now mind you the method the data and the grader all of it was open source by Zomi with their MIMO version 2.6 release. I took their music

00:33 environment one rented GPU about 1 hour 40 minutes of training and about $12 and we went from less than 1% playable music to almost 12% that’s about 20 times. So in this video I will show you end to end how to get the system ready, get the data, train the model, get the adapter, merge it and use it. So let’s go ahead and get started. So this is the news. Zomi is just open sourced around 7,000

01:04 RL environments, reinforcement learning environments that they use to train Mimo across code, cyber security, general knowledge work, web design, and music. And people are calling this one of the biggest things that happen in open-source RL environment data. And I do really agree with that. Now, MIMO version 2.6 come in two sizes. The pro is a 1 trillion parameter mixture of experts model with 42 billion active parameters and the flash one is 310

01:37 billion 15 billion active. But more than the models, the really interesting thing here is what else they released. So if you go to hugging face, you can see this data set. Memo version 2.6 RLS. It has five subsets here. So code, cyber, general, webdev, and music. And these are not just prompts. These are full environments with verifiers, which means for every task there is a way to automatically check whether the model did it right. And then we have the

02:08 training code as well. So they forked ver which is a bite dense RL training library and they added their recipes on top of it. The way um I like to think of this is bite built the school building and Xiaomi wrote the syllabus the exams the examiners and the anti-cheing rules and then they gave all it away. They also released a small 9 billion model uh Mimo version 2.6 six distill 9 billion as a starting point for your own reinforcement learning and the tech

02:41 report uh they even tell you the bill they spend about $2.6 $6 million on RL for the pro model and about $900,000 for the flash model. And the main idea of this whole paper is really simple. If you keep scaling reinforcement learning, more compute, more environments, and better graders, the model keeps getting better. You can see all the benchmark lines go up as the training steps go up. So, I wanted to test this idea myself on a small budget and I picked the music

03:13 environment and let me tell you why. music is single turn. There is there are no Docker sandboxes, no AI judge and the grader is pure Python and it’s the easiest one to run on one GPU and also you can actually hear the results which is a lot more fun than looking at code test results. So before you go into the code, let me quickly explain how this training actually works because it’s really simple once you see it. We give the model one prompt. For example, composed a calm shallow piece in E minor. Now instead of one answer, the

03:45 model writes eight different answers for the same prompt. You can see these eight attempts here. Then the grader scores every one of them. Some got zero, some get a good score. And now here is the key thing. We don’t compare the answers with some perfect answer. We compare each answer with the average of its own group. So if an answer is better than its siblings, we push the model to do more of that. And if it’s worse, we push it away from that. And that’s it. That’s GPRO or group policy or group relative policy optimization. Repeat this for

04:19 many prompts and many steps and the model slowly learns what the grader likes. Now Zomi added a few tricks on top of this and one of them I have used here as well which is the length penalty. So if two answers are both correct and one way is and one is way longer than the other in its group the one that one gets a small penalty. So the model learns to be correct and short. It’s equation four in the paper and it’s about 10 lines of code which I will show you. And one important thing that the paper says which we will see in

04:53 action in just a moment. RL can only improve what the model can already do sometimes. If the model never gets a single answer right, every score is zero. Every answer is equal to the average and there is nothing to learn from. Keep this in mind because this is exactly what happened to me. So what is this ABC notation? It’s a way to write music as plain text. So you can see here X is the tune number, T is the title, M is the meter like 3x4, K is the key, and then the notes are just letters. So C, D, E, F, G, A, B, that’s seven notes of

05:27 the 12 notes. And then we have the sharps and the flat notes as well. So total of 12 notes. And this is really good for language models because they only write text anyway. Then we can convert this text into a MIDI file and uh uh with a tool called ABC to MIDI and then into audio file so that we can listen. Now Zami’s grader there does something really smart here. It takes the MIDI and it measures 18 different things group into six areas like rhythm, texture, acoustics, tonal, register and structure. And it compares each of them

06:01 against real human music. So basically it’s asking does this sound like a human wrote it? And the final score is between one and zero. And one strict rule, if ABC has errors or too many broken bars or even a blank line in the middle, the score is straight zero. Remember this a blank line thing because it actually becomes really important. So what I did is I rented uh an 880GB GPU on RunPod. You can go ahead and check the links. Set everything up. And first I tested the base models before training anything. And this is where it

06:34 got interesting. the Quen 2.53 billion model 0 out of 160 0% and you can see why look at the output meter 203x4 and then just E flat E flat E flat E flat that’s not music grand 2.57 billion again 0%. Then I tried Xomi’s own model, the Mimo distill 9 billion and that was really surprising. It thinks so much that it used the ent 8,000 token budget yet just thinking and it never even

07:05 started writing the music. Mind you, the Xiaomi’s own training they allow 100,000 tokens for this. When I gave it 16,000 tokens, I got 2 and a half%. And the best one was quen 38 billion was around 6% with thinking on and around 1% with thinking off. So now we have a problem. Remember what I said. If everything scores zero, RL has nothing to learn from. So I went inside the failures to see what’s going on. And you can see the music itself was often fine. The problem was this blank lines. The model puts an empty line between sections like a human

07:39 would do in a document. But in ABC notation, a blank line means the song has ended. So the greater throws the whole thing out. 74 out of 80 failed pieces had this. And when I just remove the blank lines, the valid rate went from 1% to 16%. So the model can write music, it just has a bad habit. So what I did during training only, if the piece fails only because of blank lines, I grade it with the blank line removed. then it gets only 30% of the credit. So

08:10 the model gets a signal but it clearly told but it’s clearly told that you would get fixed that you would get more uh if you fix this. And I really had to be clear here for the final result that I will show you. There is no help at all. It’s strict grider unchanged on 40 prompts that the model never saw during the training. So let’s go ahead and look at the code. Everything in in is in this music demo folder and I will share this with you. First the data. This is Xiaomi’s music subset on hugging face. 10,000 prompts. Half of them are in

08:43 English, half are Chinese. So in prep data.py I take 496 English ones and I keep 40 aside as a test set. The model never trains on these 40 and that’s how we’ll check its honesty at the end. Then this MIMO scorer folder there’s a Xiaomi’s grader copied exactly from the repo. I haven’t changed a single line. You can see the six groups here with their weights. Rhythm is 22% texture 20 and so on. And then we have pre-check.py. Uh

09:14 now why do we need this? Xiaomi’s own config has a warning. If ABC2 MIDI is not installed, every score silently becomes zero and the training still looks healthy. So before spending any money on GPU this run uh this runs one handwritten walls through the grader it scores 619 and plain text scores zero. So we know the grader is live. Then this reward.py file which is the heart of it.

09:45 The first function is a pure Xiaomi grader which is used for evaluation. And then there is one training reward with the blank line credit that I just explained. And then down here we can see the length penalty which is Xiaomi’s equation 4. It looks at all the correct answers in a group takes the typical length and if an answer is way longer than it is then that it cuts a little bit of the reward. Then we have the GPRO.py. Uh we are using TRL from hugging phase with VLM for fast generation and these

10:18 are important settings. Eight answers per prompt, eight prompts per step, 30 steps, no KL penalty, which is the same as Xiaomi’s recipe. We’re using low rank adaptation or Laura, which ranks 64 with rank 64, which means we don’t train the full 8 billion weights. Uh we train a small adapter on top. This is what this fits on one GPU only. And then we have the proofs tool. Generate.py Pi runs the model on the 40 test prompts before and after with the same seat. Render.py

10:50 turns the ABC into MIDI audio and these piano roll pictures. Charts.py makes all the charts and uh blindfare.py creates a blind listening test which I will show you at the end. And to run everything, you just need two commands. Setup pod.sh installs everything and runs the pre-check. And run all.sh SH does the full test before test uh training after test audio and charts. Now for running this I use run pod uh 1H 1800 with 80GB which is about $350 an

11:22 hour. So once a pod is ready, I copy the folder uh run setup pod and you can see that it installs ABC to MIDI fluid sync TRL and VLM u download ZOMI’s prompts and then pre-check wall scores 619 garbage text score zero scorer is live really good and I’ll be honest it didn’t work for the first time the first run crashed because the setting was renamed in the new TRL version then it run out of ran out of GPU memory because eight long answers at once was too much. So I

11:57 reduced it to two answers per pass then it was working but it was too slow about [snorts] 7 minutes per step. I reduced the prompts per step and then it got about 3 and 1/2 minutes per step and now it’s training. You can see every step it prints the reward. Step one the reward is 013 and it slowly starts going up. So 30 steps took about 1 hour 40 minutes and the total bill including the pod where I had SSH key issue was about $12 for the same idea that Xiaomi spent

12:28 millions on his full scale. Of course at a much much smaller scale but the method works and this is how it’s done. So let’s see the results. First the training reward. You can see it started at point at 013 and by the end it’s around.1 sometimes.15. So that’s about 8 to 10 times higher and you can see it’s going up steadily. The model is clearly learning something. But the training reward is not the real test. So here is the real test. 40 prompts the model never saw. Four

13:01 answers each. Zami strict greater no help. Before training, 1 out of 160 was playable. 0.6%. After training, 19 out of 160, 11.9%. That’s about 20 times more playable music. And it went from one prompt of the valid piece to 13 prompts out of 40. And uh look at this one. The average answer length went from about 2,100 tokens to about 1,250. That’s 40% shorter. That’s the length penalty working just like Xiaomi described in

13:35 the paper. Now I want to be honest here 12% is still low. Most answer will uh still have the blank line habit. So this is a really good start not a finished composer but this is 30 step on one GPU. If you train it longer or first fine-tune it on good examples uh like Zami does you will definitely get better results. Now one more thing the grader measured it uh if humanlike. The grader measured if it sound humanlike not if it sounds beautiful. So I always uh also

14:08 made this blind test. So every pair has one piece from the base model and one from the train model and you don’t know which is which. I’ll put this in the description. Try to guess which one is trained and let me know in the comments. Now we get at the end of the training uh is a Laura adapter. It’s about 700 MB and it contains everything the model learned. So what I did is I merged this adapter back with quen 38 8 billion and I’ve uploaded the full model on my hugging face uh and I also converted it into GGUF format so that you can run it

14:41 on your own laptop as well. So if you go to prompt 48 on hugging face you can see Gwen 3 of 8 billion MIMO music GPRO. You can see all the results here and the adapter as well if you want to merge it yourself. And for the GGUF, the Q4 KM version is about 5GB, which will work on an 8GB GPU laptop just like mine. So you can go ahead and run this one line in Olama. And let’s give it a prompt. Compose a short calm piano base in C major. Add slash no think think at the

15:11 end because we train it with thinking of. And you can see it’s writing the ABC here. And then we convert it >> [music] >> and play it. So in this video we have seen how we can take the RL environment that Zami opens sourced and train our models to write music with GPRO on one GPU. We went from less than 1% playable music to almost 12% in about 1 hour 40 minutes for about $12. And the most

15:43 important learning here which Zomi also said in their paper, RL can only improve what the model can already do sometimes. That’s why finding the blank line problem was the real key. Now if you want to push this further uh train on it longer, try Zomi’s 9 billion with a bigger token budget or try the code and webde them webdev environments. All the scripts are in the description. The model is on my hugging face. And if you’re interested, you can go ahead and check this one out and try this out on your own system and let me know how it

16:15 goes. I will see you in another one. Check out this next