Nemotron 3 Diarization with Parakeet Locally: Who Spoke When

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Nemotron 3 Diarization with Parakeet Locally: Who Spoke When · Channel: Fahd Mirza · 08:18 · Watch on YouTube

Topics: AI models, Local and open-source models, Multimodal models

Summary

The video installs Nvidia’s Nemotron 3 diarization model locally and tests it on CPU and GPU. The presenter demonstrates speaker labeling on Hindi audio, compares offline and low-latency modes, and reports speed and VRAM results. They also test multilingual clips and show how to combine Nemotron 3 with Nvidia’s Parakeet model to produce a transcript with speaker labels. The video concludes with an invitation to subscribe to the presenter’s newsletter and support the channel.

Key points

Nemotron 3 is presented as an open-weight diarization model that identifies when speakers talk, supports up to eight speakers, and can handle overlapping speech.
The presenter installs Nvidia’s NeMo toolkit with its ASR extra on Ubuntu to access the speech models.
The first test uses a Hindi clip with two speakers and shows segment start times, end times, and generic speaker labels.
Offline mode processes larger chunks for accuracy, while live mode returns results after small chunks for lower latency.
The presenter reports VRAM use of just over 1 GB and says the model can also run on CPU.
The benchmark on an 11-minute file reports an RTFX of 390 for offline mode and 8 for live mode; the presenter says both results are faster than real time.
The video reviews leaderboard and error charts, describing Nemotron 3 as performing well on missed speech and speaker confusion.
Tests on short clips in multiple languages return one speaker for each clip, as expected by the presenter.
Combining Parakeet for timestamped words with Nemotron 3 for speaker labels produces a transcript for a Spanish two-person audio clip.

Chapters

Start	Chapter	Summary
00:01	Introduction and local installation	The presenter introduces Nemotron 3’s diarization capabilities and installs the NeMo toolkit on Ubuntu.
02:20	Speaker test and performance modes	The Hindi test output shows two speaker labels, followed by an explanation of offline and low-latency modes and a benchmark setup.
05:31	Benchmark results and multilingual tests	The presenter reports timing results for an 11-minute file and tests the model on short clips in multiple languages.
07:13	Combining Parakeet with Nemotron 3	The presenter demonstrates a Spanish transcript with speaker labels created by combining Parakeet’s timestamps with Nemotron 3 diarization, then closes the video.

Quotes

It answers who spoke when in an audio recording for up to eight speakers. — 00:01

So it tells us who spoke when not who they are. — 02:20

Offline mode scored 390 finishing in under 2 seconds. — 05:31

So parakeet is going to write the words with timestamps and Neotron says who was talking at each moment. — 07:13

Concepts and entities

Nemotron 3 Diarization, NVIDIA, NVIDIA NeMo toolkit ASR, Parakeet, Hugging Face

Full transcript (original language)

00:01 Today Nvidia released much anticipated Neotron 3 diization an openweight model with 100 million parameters. It answers who spoke when in an audio recording for up to eight speakers. It works live or on recorded audio, handles overlapping speech and can start in about 320 millisecond. We are going to install it and test it out locally. I will also delve into its architecture. But for

00:32 now, let me get it installed on my Ubuntu system. This model can run easily on CPU, but I will also show you how to run it on GPU. So, you can see that I’m installing Nemo toolkit ASR. This is Nvidia’s open-source speech AI library, and the ASR extra adds the speech models, including the diization one. Now our Nemo toolkit is installed. For the first test, I’m going to use

01:02 this script which is loading the Neotron 3 dorization model from hugging face. Then it is setting the streaming settings to the offline mode which uses a big 30-cond buffer for best accuracy. Finally, it runs the model on a two speakeraker Hindi clip and prints each segment as start time, end time, and speaker number. Let me first play that Hindi script.

01:46 customer service. So this is a two speaker one. It is around a minute long. So let me play that script and the model has been downloaded and it is working through that script. I will explain this output in a short bit. Let it finish and then we will go from there. So you can ignore these numbers. These are just startup noise of Neotron 3 model, its architecture and all that stuff. This is a real output which starts from here. So you can see that each line here has a start time and end

02:20 time and a speaker label and it has found two speakers, speaker 0 and speaker 1. And the labels are generic. So it tells us who spoke when not who they are. Speaker zero started first then they take turns in one spot both talk at once which is quite good. Now um let’s check it against a few more audio and let’s do a few more tests. Okay, next test is more realistic. This is where we are going to run a one 11 minute audio file through the model in

02:53 two modes as you can see on your screen. So I’m also benchmarking it. There is offline offline mode which waits for big chunks of audio and it is going to finish the whole file uh in longer time and then we have a live mode which is for low latency live calls. It answers after every tiny.3 second chunk. As you can see here this is for the low latency this is for the offline. So if you are using it uh for diorization for your use case if your use case is for low latency

03:23 live call use this setting otherwise use this second like a real call. Now let me take you my terminal and run this benchmark. It is going to load the model and run it in both modes and we will check out the timings afterwards. So let’s while it runs let me also show you the VRAM consumption. So this is all it is using at the moment.

03:54 So it’s not much really just over 1 gig. You can easily run it on your CPU. Okay. So you see it is uh the offline mode which is running at the moment. And while it does this benchmarking, let me show you few of the benchmarks and other information about this model. So this one is voice arena leaderboard lower bars are better and neotron 3 diorization is the shortest bar on the bar left. So it makes the fewer mistakes and you can also notice next best open

04:29 model is Nvidia’s own previous one and the new model clearly beats it. Big names like 11 Labs, Meta and Assembly AI sit much much higher. Look at this one. This table shows the full picture. Neotron takes first place and it is open source which is rare at the top. The column breaks the errors into types like missing speech, false alarms and mixing of speakers. Look at the speaker count columns too. They show how often each model guessed the numbers of speaker

04:59 right. Finally, they have also shared this. This one shows what kind of mistakes each model makes. Orange is missed speech, blue is false alarm and green is speaker confusion. Many transcription services are mostly orange, meaning they skip a lot of speech because diization is a side feature for them. Neotron has very little orange and very little green. So it rarely misses speech and rarely mixes up who is talking. So this chart 3 claim

05:31 um is quite high. But we are also going to test it out on some of the low quality audio. And here is the result. I have also just minus the noise and reran this and you can see the result. So this is an 11 minute file in two modes. RTFX means how many times faster than real time the model runs. Offline mode scored 390 finishing in under 2 seconds. Live mode scored 8 taking 79 seconds but that still keeps up with the live call since

06:03 anything above one is real time. Nvidia’s own numbers are higher but I think they used a different GPU in settings so the pattern matches even if the exact numbers differ here and now let’s test out multilinguality with one uh with many short clips in many languages as you can see on my directory on the local system for each clip what I’m going to do to use this script it is going to go with every script in this loop and then going to tell us how many speaker it found and how any speech segments. So should be

06:38 fun as it contains many languages from Udu, Hindi to European languages, African languages and few others including Arabic etc. And there you go. It understood every language and every clip came back as one speaker across all languages which is what we expected if each clip is one person talking. And this is actually uh quite good because not only the speaker, it has also given us the segments and each and every language out there.

07:13 And here is the final and most interesting test. I’m going to combine two models. Nvidia’s parakeet with this new three dorization which would be the real world use case. So parakeet is going to write the words with timestamps and Neotron says who was talking at each moment. And this is the result. I already have just run it. You can see the result is a transcript with speaker labels. It is a twoerson uh audio in Spanish. So let me quickly now play that clip and you can also

07:47 check it out a little bit. You see this is how you can use this model in your own real world use cases.

08:18 So that’s it about this new model. I also have recently started this free weekly AI newsletter which you can subscribe at bajmeza.ubstack.com. And if you want to help out the channel, please become a member. Thank you for all the