Voice of Reason from Kyutai on Top of GLM: Run Locally

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Voice of Reason from Kyutai on Top of GLM: Run Locally · Channel: Fahd Mirza · 07:53 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI models, Local and open-source models, Multimodal models

Summary

Voice of Reason is presented as a 9-billion-parameter speech-native model that reasons about spoken math without first converting speech into text. The video explains its reinforcement-learning approach, installs and tests it on an Ubuntu server with an Nvidia H100, and reports that the model uses about 20 GB of VRAM during testing. It answers several spoken math questions correctly, but makes a mistake on a harder problem by combining rates when the dogs were working independently. The presenter says the model was fine-tuned on 150,000 spoken math question-answer pairs and improved from 27% to 74.8% on a math benchmark.

Key points

The model takes spoken questions and performs its reasoning in the spoken domain, rather than relying on a speech-to-text-to-speech process.
The model card reports a 74.8% score on GSM8K, which the presenter describes as a state of the art for speech-native models.
The presenter installs the model on Ubuntu with an Nvidia H100 GPU with 80 GB of VRAM.
The described reinforcement-learning process generates multiple attempts, uses a judge to score them as right or wrong, and uses that feedback to improve the model.
The test script takes spoken audio, feeds it to the model, and displays its reasoning and answers as text.
The first model download is described as just over 19 GB; the presenter observes VRAM use of about 20 GB.
The model handles the tested multiplication questions and an equation question successfully.
On a multi-step problem about dogs working independently, the model incorrectly combines their rates and gives 18 instead of the correct answer, 19.
The presenter says the model was built on GLM-4, fine-tuned on 150,000 spoken math question-answer pairs, and then trained with reinforcement learning.

Chapters

Start	Chapter	Summary
00:01	What makes Voice of Reason different	The presenter introduces the speech-native model, its reported GSM8K score, and the plan to install and test it.
01:06	Installation and architecture	The presenter sets up the model on an Ubuntu server and explains how spoken questions, multiple attempts, and reward feedback support its reasoning.
03:11	Running spoken math tests	The presenter runs the voice-in, text-out test script, demonstrates multiplication questions, and checks the model’s VRAM use.
05:18	A harder test and the training pipeline	The model solves an equation but makes a rate-combination error on a harder problem, after which the presenter describes its fine-tuning and reinforcement-learning process.

Quotes

It reasons entirely in the spoken domain. — 00:01

An LLM judge then looks at all those attempts and simply asks, is this right or wrong? — 02:07

That’s it. — 04:15

So, the correct answer is 19 drugs, not 18. — 06:21

That is what good post training looks like. — 07:53

Concepts and entities

Voice of Reason, Kyutai, GLM-4, GSM8K, reinforcement learning, LLM as a judge, NVIDIA H100, eSpeak, Ubuntu, Mass Compute, Fahad Mirza

Full transcript (original language)

00:01 This is Voice of Reason, a 9 billion parameter speech native model from Kyoto AI that can actually do math. What makes it special is that it reasons entirely in the spoken domain. No cheating by converting to text first, thinking, then converting back. It uses reinforcement learning with a verifiable reward signal exactly like Deep Sea Car One did for text, but applied very to a model that speaks and listens natively.

00:34 The result, 74.8% on GSM8K as they have shared on their model card, which is a new state-of-the-art for speech native models. So, today we are going to pull this model down, set it up on a fresh Ubuntu server, and ask it some math questions out loud and hear it reason through the answers in its own voice. This is Fahad Mirza, and I welcome you to the channel. I will also unpack its architecture, but for now, let’s get it

01:06 installed on this Ubuntu where I have one GPU card, Nvidia H100 with 80GB of VRAM. Now, you don’t need to buy it. All you need to do is to just go to the video’s description, get a link of Mass Compute, and you can rent this GPU on a very affordable price. Also, recently I have started this free weekly AI newsletter, which you can subscribe at fahadmirza.substack.com. Coming back to this model, let me quickly get clone the

01:37 repo, and I will drop the link to it in video’s description. Also, I’m going to install this version of torch with transformer. Make sure if you are following along, you install exactly these versions. While it gets installed, let me unpack the architecture or what is actually happening under the hood. You take a math question, something like what is 4 * 7? And instead of typing it, you speak it. That audio gets converted into tokens,

02:07 which are basically little chunks the model can understand. The model then generates a group of answers, not just one, but several attempts. An LLM judge then looks at all those attempts and simply asks, is this right or wrong? The ones that got it right get a thumbs up. The ones that got it wrong get a thumbs down. That feedback is then used to update the model, basically telling it do more of what worked. Over thousands of examples, the model gets better and

02:38 better at reasoning through spoken math problems. No human labels, no hand-crafted rules, just trial, error, and a reward signal. That is reinforcement learning in a nutshell, and that is what is being used by this model, but in the modality of voice. So, this is what this model is doing. Let me take you back to the terminal, and by the way, if you want to help out the channel, please become a member. The link is on the home page of the channel or in the first pinned comment.

03:11 Everything is installed. Now, let’s do the testing. This is a script which I’m going to use for testing. It takes a spoken audio question, converts it into tokens the model understands, feeds the to this voice of reason model, and prints out the model reasoning and answers as text. So, this is voice-in, text-out math tutor. So, you speak the question, it thinks out loud, and gives you the answer. First time when you run this, it is going to download this Q time model. So,

03:43 let me go back and run the script. The model is being downloaded, and the size is just touch over 19 gig. And while it runs, let me play that audio, which is just a robotic voice. >> What is 15 multiplied by eight? What is 15 multiplied by eight? >> There you go, and it has given us the answer. Um and I just generated that from eSpeak, and you can see that it worked fine. Let me run this again.

04:15 And I’m going to show you the VRAM consumption. So, you see the jumping up. And going up at just around 20 gig of VRAM. That’s it. So, that is a VRAM consumption in order to get this answer. Let’s do a few more tests. And if you’re wondering how exactly I’m generating those robotic voices, it is just with eSpeak. You see, I just generated it this what is 12 multiplied

04:47 by 12 in the question.wav. Let me also play that, and we will run that script again. So, I’m going to just play it now quickly. Just give me a sec. I’m finding where exactly that file is. >> What is 12 multiplied by 12? >> There you go. So, let’s run this. And look at the speed, too. It is quite quick. I will just let it run so that you could also see how long does it take.

05:18 So, this is our answer, which is correct. Okay, let’s do another one in my own voice. Solve this math equation. x + y is equal to x - y. Let’s wait for the model to come back. And there you go. I think the model has done well. It has given us all the steps which would be used in order to solve this math equation question. Uh pretty good. Very targeted, very

05:49 nuanced. I think they also have another flavor where you can use it with lot of reasoning. Anyway, let’s do another quite a hard one. Basically, it would be a sort of stress test. So, you can see that I’m just generating that um audio with this prompt where this is real-world, multi-step, involves parallel work streams. So, I think this is a good stress test for this model. So, let’s see how it goes with this new audio which I’m saving in question.wave

06:21 file. So, from the response, you can see that the model reasoned through it step by step correctly, breaking down individual rates, finding a common denominator, and converting 3 hours to 180 minutes. But, I think it misread the question. If you go at the top and look at the question, the dogs work independently on separate drugs, not together on one. So, the correct answer is 19 drugs, not 18.

06:51 But, the model’s combined rate approach is a classic mistake humans make on this exact problem, too, which actually makes it uh quite challenging for the model. So, I would say despite of being um it using the GLM underneath, it failed this test. Before I wrap up, one thing I found really interesting from their paper um because that was quite clever how they built on top of GLM-4 uh rather than starting from scratch. They took the existing 9 billion

07:22 parameter model, first fine-tuned it on 150,000 spoken math questions, answer pairs to get it comfortable with the domain, and then unleashed reinforcement learning on top. During RL, the model generates multiple attempts at each spoken question. A judge model simply scores each one right or wrong, and the model learns to do more of what works. So, as I said earlier, no human labeling, no handcrafted rules. The base GLM for voice scored just 27% on math

07:53 benchmark. After their pipeline, it jumps to 74.8%. That is what good post training looks like. So, a very good exercise in post training if you’re interested in that field. Check it out. Let me know what do you think. Again, if you want to support the channel, please become a member and subscribe to this newsletter. Thank you for all the support.