I Got Real Speech Recognition Running on a $10 Chip (Whistle)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: I Got Real Speech Recognition Running on a $10 Chip (Whistle) · Channel: Better Stack · 12:01 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, Local and open-source models, Multimodal models

Summary

The video demonstrates Whistle, a compact speech-to-text model, running fully offline on an ESP32-S3 microcontroller to control its built-in RGB LED using spoken commands in multiple languages. Because the 16.9 MB model exceeds the board’s 16 MB of flash, the creator trims its decoder to four layers and optimizes inference to fit and run faster. The board processes commands in roughly 10 seconds on average, with reported accuracy of about 78% for English color commands in tests using synthetic voices. The video also explains the model’s architecture, hardware setup, performance optimizations, and techniques used to improve recognition.

Key points

Whistle is a 16.9 MB speech-to-text model from Cactus Compute that supports English, German, French, Spanish, Italian, Dutch, and Polish.
The model can identify the spoken language, provide word-level timestamps, and generate embeddings without producing a transcript.
The ESP32-S3 has 16 MB of flash, so the creator removed decoder layers to fit the model; the four-layer version leaves 98 KB to spare.
The encoder retains all eight layers, while the decoder can be reduced to fewer layers.
The prototype uses an ESP32-S3 board, an INMP441 digital microphone, a push button, and the board’s RGB LED.
The model converts audio into a mel spectrogram, encodes it, and generates a transcript one piece at a time.
Processing was optimized by handling multiple audio frames per pass, splitting work across both cores, and using 8-bit integer math on the chip’s vector unit.
The four-layer model takes about 10 seconds on average; example commands took 8.8 seconds in English and 11.7 seconds in French.
In the creator’s synthetic-voice tests, English color commands were recognized correctly about 78% of the time; short phrases performed better than single color words.
Keyword biasing and a retry rule helped the trimmed model outperform Cactus’s full-size model on the creator’s color-command test.

Chapters

Start	Chapter	Summary
00:00	Offline voice-controlled LED demo	The creator demonstrates an ESP32-S3 recognizing English and French commands to control an LED without Wi-Fi or Bluetooth.
01:02	Whistle and fitting the model on the chip	The video introduces Whistle’s capabilities and explains how trimming decoder layers lets the model fit in the board’s limited flash.
05:15	How speech recognition works and hardware setup	The creator walks through the audio-processing pipeline and describes the microphone, button, board, and LED setup.
07:52	Live tests and recognition performance	English and French commands demonstrate the system’s response times, and the creator explains why short phrases work better than isolated words.
09:24	Optimizations, accuracy, and wrap-up	The creator outlines inference optimizations and accuracy-improving techniques, then summarizes the project and points viewers to Whistle.

Quotes

it’s running completely offline with no Wi-Fi and no Bluetooth connected. — 00:00

the whole model is just a single file of 16.9 MB. — 01:34

that leaves me with a grand total of 98 kilobytes to spare. — 04:12

short phrases were recognized correctly 82 to 93% of the time — 08:54

I got it running completely offline on a $10 chip. — 11:29

Concepts and entities

Whistle, Cactus Compute, ESP32-S3, Needle, Whisper base, LibriSpeech, INMP441, mel spectrogram, keyword biasing, Apache License 2.0, Hugging Face

Full transcript (original language)

00:00 Okay, folks, check this out. This is an ESP32-S3 microcontroller, and this is a tiny microphone soldered onto it. And this is a little button soldered onto a perf board that activates the microphone. And the board itself is attached to a power bank, so it’s running completely offline with no Wi-Fi and no Bluetooth connected. Now, check this out. Flash the light purple for 5 seconds.

00:31 Look at that. How cool is that? Now, let’s try a different language. Allume la lumière en rouge. Let’s see what happens. Look at that. We’re getting a red light. How cool is that? So, the model doing all this is called Whistle, and it’s the brand new speech-to-text model from Cactus Compute, the same company that released Needle, which I’ve already covered twice on this channel. So, in

01:02 this video, we’re going to take a look at Whistle, and I’ll show you how I squeezed it into a tiny chip that is technically too small for it. And then, I’ll show you some pretty weird things I discovered along the way. It’s going to be a lot of fun, so let’s dive into it. So, a quick bit of context here. In my previous Needle 2 video, I wrote my own inference engine to get Needle running on this exact board, and it responded to my commands that I had to type out by hand. But, Whistle now gives us the

01:34 opportunity to deploy a fully local speech-to-text model on a microcontroller. So, Cactus released Whistle on October 2nd, and the whole model is just a single file of 16.9 MB. It transcribes English, German, French, Spanish, Italian, Dutch, and Polish, and it figures out which language you’re speaking on its own. It can also give you a timestamp for every word, and it can turn a clip into an embedding without writing out a transcript at all. For comparison, OpenAI’s Whisper base

02:05 model is 145 MB, so Whistle is almost nine times smaller, and according to Cactus’ own benchmarks, it still beats Whisper base on LibriSpeech, which is the standard test set of people reading audiobooks out loud. On a MacBook M4 Pro laptop, it gets its first word out in about 11 ms, which is blazingly fast. And the part that makes it interesting for us is that Whistle runs on the same engine as Needle, and Cactus designed the two to be loaded side by side. So,

02:38 in their setup, you can feed an audio clip straight in and get a tool call straight out, which is basically a voice assistant for tiny devices. Now, Cactus ships pre-built versions of its engine for 17 platforms, including phones, laptops, and a browser, but there’s no build for the ESP32 chip I’m using here. So, just like with Needle 2, if I wanted something on this board, I had to build it myself. So, here’s how I did it. So, the first problem is simple. The board has 16 MB of flash storage, and Whistle

03:10 is 16.9 MB. So, the full model is literally bigger than the chip itself, and that’s before we even count the firmware. But, this is where the intelligence ladder comes in, which some of you will remember I covered in the Needle 3 video. So, Whistle has two halves. There’s an encoder, which listens to the audio, and there’s a decoder, which writes out the text. The decoder has eight layers, and Cactus trained it so that you can cut it off after two, three, four, or more layers,

03:40 and every one of those cut down versions still works as a model on its own. But, the encoder always runs with all eight of its layers, so we cannot trim that part down. So, I wrote a small tool that takes the full model file and writes out a new one that only contains the layers you actually want to keep. And I checked that the strip file gives exactly the same output as the full model running at the same depth, and I made sure that Cactus’ own engine loads without any complaints. And when I printed out where all the megabytes actually go, the

04:12 biggest chunk was actually the N-gram memory, and those are the big lookup tables I talked to about in the other needle videos, which take up about 5.3 megabytes. So, that’s almost a third of the whole file. One decoder layer, by comparison, is only about half a megabyte. So, I went with the four-layer version, which fits quite nicely once I stretched the model partition all the way to the end of the flash chip, and that leaves me with a grand total of 98 kilobytes to spare. And it also makes noticeably fewer mistakes than the

04:44 two-layer version. So, those extra layers are definitely worth packing into it. The only catch is that needle and whistle can’t share the same board, because together that would need about 24 megabytes. So, whistle is running on its own, and instead of needle, I wrote a small parser that reads the transcript and pulls out the color and how the lights should blink. So, let’s quickly walk through what actually happens between me speaking and the text coming out, because it helps explain where all the latency goes. First, the raw audio

05:15 gets turned into something called a mel spectrogram. The microphone records 16,000 samples per second, and the model chops that into tiny 10-ms slices. And then it measures how much energy there is at each pitch. So, you end up with something that looks like a heat map of your voice over time. And then a small stack of convolution layers squeezes that heat map down so that every slice now covers 80 ms of audio instead of 10. And that’s important because from this point on, every part of the model works

05:47 on those 80 millisecond frames. So, what 2-second command becomes just 25 frames. And next comes the encoder. It has eight layers, and the interesting thing here is that every frame gets to look at every other frame in the clip at the same time. So, a sound at the start of your sentence can be understood in the context of what you said at the end. And finally, the decoder writes out the transcript one piece at a time, kind of like a regular language model. The very first thing it writes is a language

06:19 token, so it literally decides whether you’re speaking English or French before it writes a single word. And every time it writes a piece of text, it looks back at the encoder’s output through something called cross attention, which is basically the decoder checking back with what it heard before it decides on the next word. The decoder also has those engram lookup tables from needle, so some of its knowledge comes from looking things up instead of calculating them. So, now let’s look at the hardware. It’s the same kind of ESP32-S3

06:51 board I used in my Needle 2 video with 16 MB of flash and 8 MB of memory running at 240 MHz. On top of that, I soldered on an INMP441, which is a tiny digital microphone that costs a couple of dollars. And I also soldered together a little perf board with a regular push button, which lets me speak into the microphone when the button is pushed. And in total, that comes down to eight wires I had to attach to the board. And the light I’m

07:21 controlling is the RGB LED that’s already built on the board. And just like last time, I’ll leave the link to the GitHub repo in the description below if you want to build this gadget yourself and flash it with this exact project. So, the way it works is that you hold the button while you speak, and the LED turns white, so you know it’s recording. When you let go, it pulses dimly while it’s thinking, and then it does whatever you ask for. And if you connect the board to a laptop, I have this terminal app that shows the transcript, what the LED is doing, and

07:52 how long each step takes. The laptop is only showing the output here, but all the actual work happens on the board. So, I have my board connected here, and now if I press the button, we should see something on the TUI application. So, let’s start talking. Flash the light blue. So, now it’s transcribing. And you can see flash the light blue, as well as you can see it on the board. And

08:23 you can see it took it a total of 8.8 seconds to process this command. Let’s try something in French. Fais clignoter la lumière violette. And it’s transcribing. And as you can see, we get a purple light here, and also on the app, and it took us 11.7 seconds. So, as you can see, it does vary. Some commands take a longer time to respond, some are shorter. So, it

08:54 does depend on the prompt, the language, everything combined. And if you remember in my Needle 2 video, a single command took around 40 seconds. So, this is already a lot snappier. One tip, if you build this yourself, say a short phrase like make it red instead of just saying the word red, because in my tests, short phrases were recognized correctly 82 to 93% of the time, while a single color word on its own only worked 44% of the time. And this might sound

09:24 counterintuitive, because you’d think one clear word would be the easiest to recognize, but actually, it’s the other way around. Because a speech model like this learned from people talking in full sentences, so a single word on its own gives it very little sound and almost no context to work with. Now, to get a board like this to respond in such a quick manner took some work because the very first version took almost 24 seconds for a 2-second command, and most of the processing time in a model like

09:55 this goes into the matrix math, which is basically doing millions of multiplications between the audio and the model’s weights. So, I optimized that part so that it processes several audio frames in one pass and splits the work across both of the chip’s cores. And then I moved the heavy math to 8-bit integers on the chip’s vector unit, which is a small piece of hardware inside the ESP32-S3 that can do several multiplications at the same time. And with all of that, I got the four-layer model to respond in

10:26 about 10 seconds on average. So, I think that’s a pretty nice result for this experiment. Now, let’s talk about transcription accuracy, and I want to be up front that these numbers come from synthetic test voices and not from real people, so take them as a rough guide. But in my tests for English color commands, the board gets the command right about 78% of the time, and that barely changes when I’m mixing real noise from my microphone. And the funny thing is that on this color command test, my cut-down version on the chip

10:58 actually beats Cactus’s own full-size model in every language. The small model itself isn’t any smarter, and that difference comes from two tricks that I added on top. The first is keyword biasing, where I give the model a list of color words and it gets a little nudge towards them whenever it’s unsure. And the second is the retry rule, so if the first attempt doesn’t find the color, the board tries again with a different language and only accepts that answer if it’s almost as confident as the first one. So, there you have it,

11:29 folks. That is Whistle in a nutshell. It’s a 16.9 megabyte speech-to-text model that understands seven languages and with a bit of trimming and a lot of optimization, I got it running completely offline on a $10 chip. Full credit to Cactus Compute, they are the real magicians who created this awesome speech-to-text model. So, a big shout-out to them and Whistle is released under the Apache 2.0 license and it’s available to download on Hugging Face. So, you can start tinkering with it on your own as well.

12:01 So, what do you folks think about Whistle? Have you tried it? Will you use it? What kind of projects would you build with it? I would love to know your thoughts in the comments section down below. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video and also don’t forget to subscribe to our channel. This has been Andras from Better Stack and I will see you in the next videos. >> [music] [music]