This 164MB Speech Model Claims to Beat Whisper... I Checked

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This 164MB Speech Model Claims to Beat Whisper… I Checked · Channel: Better Stack · 09:18 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI developer tools, Local and open-source models, Multimodal models

Summary

The video evaluates Phonon 2, a compact English speech-recognition model from Fermion Research, and examines its claims of high speed and accuracy compared with Whisper and other models. It explains how the model is compressed, demonstrates local transcription and developer features, and reviews both Fermion’s benchmarks and independent leaderboard results. The presenter finds that Phonon 2 performs well, but says the speed comparison with Redux may be affected by the runtime used, and the available results do not establish that it is better in every case. The conclusion is to experiment with Phonon 2, especially for English audio on Apple Silicon, rather than switching to it outright.

Key points

Phonon 2 is a 164 MB download that expands to about 178 MB and focuses on English, unlike Redux, which supports 25 languages.
Fermion says Phonon 2 can transcribe at 174 times real-time speed on a 16 GB M5 MacBook Air using its fast path decoder; exact decoding and CPU-only speeds are lower.
In Fermion’s seven-test-set results, Phonon 2 scored a 5.21 word error rate, compared with 5.69 for Redux, 6.58 for Whisper Turbo, and 4.96 for the original Nvidia model.
The presenter notes that Fermion’s results differ from Redux’s own model card and that Redux beats Phonon 2 on the AMI meeting-audio benchmark in Fermion’s tests.
Fermion tested Redux through a third-party Fluid Audio Core ML port; the presenter cautions that this may affect the six-times-faster comparison.
Independent Open ASR leaderboard results show Phonon 2 beating Whisper Turbo on average, while Whisper performs better on three individual test sets, including noisy speech.
Phonon 2 supports microphone-based live captions, hot words, and a local server with an OpenAI-compatible transcription endpoint and a web socket stream.
The code and CLI use Apache 2.0, and the model weights use CC BY 4.0 inherited from Nvidia; commercial use is allowed with the required credit.
The presenter recommends experimenting with the new model rather than switching to it outright.

Chapters

Start	Chapter	Summary
00:00	Introducing Phonon 2	The presenter introduces Phonon 2’s size and performance claims, then explains how its English-focused compression approach differs from Redux.
01:36	Running it locally and assessing speed	The video demonstrates installation and compares Fermion’s reported transcription speeds across decoding modes and against Whisper Turbo.
04:15	Checking the benchmarks	The presenter questions the speed comparison’s runtime choice and reviews both Fermion’s accuracy results and independent leaderboard findings.
05:58	Features for developers	The video demonstrates live captions and hot words, then explains Phonon 2’s local OpenAI-compatible server and notes that streaming latency remains unverified.
08:12	Licensing and verdict	The presenter covers the licenses and project’s early stage, then recommends trying Phonon 2 without replacing existing tools yet.

Quotes

An hour of audio transcribed in 21 seconds on a MacBook Air with no internet. — 00:00

Obviously, the lower, the better. — 03:11

So, Phonon 2 still beats Whisper on average, but Whisper actually wins on three individual test sets. — 05:24

The underlying model is an offline model. — 07:07

Don’t change anything yet, just experiment with it. — 08:45

Concepts and entities

Phonon 2, Fermion Research, Parakeet Redux, Whisper Turbo, MLX, Fluid Audio, Open ASR leaderboard, Data

Full transcript (original language)

00:00 An hour of audio transcribed in 21 seconds on a MacBook Air with no internet. And the model doing it is a 164 megabyte download. Whisper Turbo, the model most people would reach for, is 10 times that size. It comes from a lab that’s barely 6 weeks old, and it claims to be more accurate than Whisper, and six times faster than its closest rival. But when I looked at who ran those benchmarks, and how they ran them, something didn’t add up. >> [music]

00:31 >> So in another video, we tested Parakeet Redux. It took Nvidia’s 2 and 1/2 gigabyte Parakeet model and compressed it down to just 178 megabytes using 1.58 bit weights. Now there’s another model called Phonon 2 from a small lab called Fermion Research, and it starts with the exact same Nvidia model, but instead of using Redux’s approach, Fermion retrained the encoder so every weight uses just one of five learned values. That’s about 2.1 bits per weight, with

01:02 everything else stored in six-bit tables. Now you’d expect that to make Phonon 2 bigger, but it doesn’t. It’s a 164 megabyte download that expands to about 178 megabytes, exactly the same size of Redux. The difference is what each model is trying to do. Redux spreads its capacity across 25 languages, Phonon 2 focuses everything on English. And according to Fermion, that tradeoff makes a pretty big difference. If you enjoy coding tools to speed up your workflow, be sure to subscribe. We have videos coming out all

01:36 the time. So, how fast is this? Well, let me show you. Getting it running is pretty darn simple. Yeah, you need Python 3.10 or newer. macOS still ships with 3.9, so you’ll want a fresh virtual environment. Then install Fermion Research with the MLX extra. All we’re going to do is run Phonon transcribe on an audio file that I’m going to execute right here. Now, Fermion claims Phonon 2 can transcribe audio at 174 times real-time

02:07 speed on a 16 GB M5 MacBook Air. That really means an entire hour of audio in about 21 seconds. And remember, this is all running locally. No uploading anything to a cloud and no paying by minute, which is a reason why we love these tools. Now, for comparison, Whisper Large V3 Turbo is about 1.6 GB in download. Phonon 2 is a fraction of that size, but there’s something important about how it gets that speed. Now, Phonon 2 uses MLX to run on the

02:37 Mac’s GPU, but it also has something called a fast path decoder, which is enabled by default. And that makes a huge difference. Fermion reports 174 times real-time with the fast path. Switch to exact decoding and that number drops to about 109 times. CPU only, about 40 times. Their benchmark also puts Whisper Turbo running through Whisper CPP with Metal at just 17 times. So, on paper, Phonon 2 looks ridiculously fast, but speed doesn’t mean much if the transcription is just plain wrong. And this is where things

03:11 start to get a little more interesting on the accuracy. Fermion tested Phonon 2 against Reducts Whisper Turbo and the original Nvidia model. They measured word error rate, which is basically how many transcription mistakes a model makes. Obviously, the lower, the better. Across seven test sets, Phonon 2 scored 5.21. Reducts scored 5.69. Whisper Turbo came in at 6.58. And the original 2.5 GB Nvidia model scored 4.96.

03:43 Think about that for a moment. A 164 MB download gets within about a quarter point of a model roughly 15 times its size. Fermion even calls it the most accurate open English speech model under 900 megabytes, which sounds impressive. A lot of testing needs to go into this though. But why specifically 900 megabytes? And more importantly, who actually ran these benchmarks? All right. Now, every single number I’ve shown you so far came from Fermion, including the numbers from Reducts. And Reducts’ own model card reported a

04:15 different average of 6.55. Now, that doesn’t automatically mean anything is wrong. They’re using different testing methods. So, you can’t directly compare the two model cards. And even in Fermion’s own results, Reducts actually beats Phonon 2 on AMI, the meeting audio benchmark. 9.16 versus 9.37. So, Phonon 2 isn’t winning everything. But remember that claim about being six times faster. This is where I started questioning the whole comparison. Fermion didn’t test Reducts using Moon Dream’s own runtime. They used a third-party Core ML port

04:49 called Fluid Audio. And that version of Reducts only reached 27.8 times real speed. But look at this. In the exact same table, the original Nvidia model, which is about 14 times bigger, reached 105 times through that same port. So, somehow, the much larger model was nearly four times faster. That’s a pretty strong sign that the port is affecting performance. Not necessarily that Reducts itself is slow. And Moon Dream reports 38 to 43 times real speed on an older M2 MacBook Air. So, Phonon 2 looks fast, but that six

05:24 times comparison doesn’t prove it’s six times faster than a properly optimized Reducts implementation. And there’s one more benchmark worth looking at. The Open ASR leaderboard team independently tested Phonon 2 and added it to their leaderboard on October 2nd. And their results tell more of a different variety here. Phonon 2 averaged 7.23. The original Nvidia model scored 6.51 and Whisper Turbo scored 8.39. So, Phonon 2 still beats Whisper on average, but Whisper actually wins on three individual test sets. The biggest one is

05:58 noisy speech. On urgent 2024, Whisper scored 13.69 compared to Phonon 2’s 14.90. It also won on SPGI speech and voice arena monsoon. And there’s another limitation. Redux isn’t even on the leaderboard. So, we don’t have an independent head-to-head between the two smallest models. Now, benchmarks are one thing, but there are a few features that make Phonon 2 interesting for us devs. First up here is live captions. Just run phonon listen and it starts transcribing audio directly from your microphone.

06:33 Here I am just talking to my microphone running a test. Now, that’s pretty useful. But, here’s another feature that I like, hot words. You can give Phonon 2 a list of names or specific words that you want to actually recognize, which is especially helpful when you’re transcribing any type of technical conversations, really anything you’re constantly repeating. Let me show you the difference. The first one is without hot words, then with hot words. Now, one thing I haven’t verified is how long the actual streaming latency is.

07:07 Those are two, one without hot words, one with hot words. The underlying model is an offline model. So, if you’re building something that depends on real-time transcription, I’d measure that latency first. But, there’s another feature that’s even more useful for us. Phonon 2 comes with its own local server. Just run phonon serve and that starts a server on port 8000 with an OpenAI compatible transcription endpoint and a web socket stream. And that’s already useful. Let’s build on that. If your application already uses OpenAI’s

07:38 transcription API, you can point the base URL at localhost instead. So, you’re running a speech-to-text locally through an API your code and you already actually understand. You can just fire up a request just like this and it’s going to execute. But, before you start building around this, let’s touch on something cuz there’s one more thing we need to look at. Because with small open-source speech models that are popping up these days, the license is going to be the biggest catch on a lot of these as we learned before. And this time though, good news, it’s actually pretty straightforward. Phonon 2’s code

08:12 and CLI are Apache 2.0 and the model weights use CC BY 4.0 inherited from Nvidia. So, commercial use is allowed. And as long as you give the required credit to Nvidia and Fermion, that’s nice. Go ahead. But, the project is still really new. As I said, it’s about 6 weeks old and it comes from a small lab that, well, kind of blew up in the last week or so. So, clearly people are interested, but the Hacker News post got only a few points and no comments. That Twitter post was the big bang. The main real-world user so far is Fermion’s own

08:45 Mac dictation app, Data, which brings us to the actual question. Should you use Phonon 2 instead of Redax or Whisper? Well, I’ll let you be the judge of that. Probably not. It’s just cool to test these models to see where they’re actually going to go, what’s going to come out on top, and what’s not. So, using Phonon 2 and experimenting with it might just be the right play here. If you’re working with English audio on Apple Silicon or you want fast local transcription with an API, this looks like a really great option. Don’t change anything yet, just experiment with it. I’m Josh from Better Stack. If you enjoy

09:18 coding tips and tricks like this, be sure to subscribe. We’ll see you in another video.