Fish Audio Drama 3: Is This Me or My Cloned Voice? Run Locally

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Fish Audio Drama 3: Is This Me or My Cloned Voice? Run Locally · Channel: Fahd Mirza · 08:16 · Watch on YouTube

Topics: Local and open-source models, Multimodal models

Summary

The video demonstrates Fish Audio’s Drama 3 text-to-speech model, including voice cloning, speaker tags, and expressive delivery controls. The presenter installs and runs it locally, noting that it requires substantial GPU memory and a consent check before cloning a voice. Several examples test emotional and dramatic speech, with mixed results and occasional need to regenerate. The presenter concludes that the model is promising but not yet suitable for production, and that curated demo clips sound better than some real-time generations.

Key points

Drama 3 is described as a 35-billion-parameter model with 3 billion active parameters per token.
It supports voice cloning, multi-speaker generation, and natural-language control, with up to 16 characters in one text using speaker tags.
The model supports delivery tags such as laughs and sighs, as well as paired tags that affect specific words.
The presenter runs the model locally using a cloned repository, prerequisites, and a serving setup.
The demonstration includes a voice-consent gate; the presenter reports a 93% match when verifying consent.
The presenter uses an H100 GPU with 80 GB of VRAM and says the hardware requirement is high.
Examples test whispering, emphasis, nervousness, sports commentary, suspense, and a fast-slow goodbye.
The presenter says generation is imperfect and may need to be repeated; performance varies across examples.
The presenter considers the model not yet ready for production and finds the curated Hugging Face clips noticeably better than some live generations.

Chapters

Start	Chapter	Summary
00:02	Introducing Drama 3	The presenter introduces Fish Audio’s Drama 3, outlines its capabilities, and discusses speed and local hardware requirements.
02:07	Local setup and voice cloning	The presenter walks through the consent check and demonstrates generating a cloned voice.
03:08	Expressive speech tests	Examples test delivery tags, emotional dialogue, match-day commentary, and suspense, with mixed results.
06:13	Limitations and curated demos	The presenter notes the model’s imperfections, compares live generations with curated clips, and assesses its readiness.
07:45	Final assessment	The presenter summarizes the voice-cloning results, hardware demands, and possible variation in output before thanking viewers.

Quotes

The voice you just heard, was that me or was that my cloned voice? — 00:02

It is a safeguard from Hugging Face that only lets a voice be cloned after the speaker explicitly agrees. — 02:07

60% it got it right and 40% I think still needs improvement. — 05:41

this is not a perfect model by any stretch of imagination. — 06:13

the consent step is a thoughtful touch. — 07:45

Concepts and entities

Drama 3, Fish Audio, Hugging Face, Vast Compute, H100, Fahad Mirza

Full transcript (original language)

00:02 Today, I will tell you how to get rich by using AI in a day. Come on, if I knew that, would I be making videos every single day like crazy? Fine. Now that I have your attention, I have a question for you. The voice you just heard, was that me or was that my cloned voice? Drop your guess in the comments. This is Fahad Mirza and I welcome you to the channel. The model in question today is this new

00:34 Drama 3 from Fish Audio. Drama 3 has 35 billion parameters, but only 3 billions are active per token because the text backbone is a mixture of expert model. It supports voice cloning, multi-speaker generation and natural language control. One text can carry up to 16 different characters using speaker tags. You can also drop in tags like laughs, sighs, right where you want them to happen as

01:05 you just saw in my first example. On speed, the model card reports first audio in under 100 ms on a single stream. Not only that, let me quickly show you that the VRAM consumption is this and make sure that if you’re running it locally, you have Hopper or Ada GPU card as you can see that I’m using this H100 with 80GB of VRAM. Yes, it takes a lot of VRAM. But if you’re looking to rent a GPU on

01:35 very good price, you can find the link to Vast Compute in video’s description. Please also become a member of the channel to keep the lights on and this is a newsletter which you need to subscribe to. This is weekly at fahadmirza.substack.com. So, how did I get it installed? I just used these three statements. I cloned the repo and then I have installed all the prerequisites and then I am serving it with HTL only

02:07 on my local system. So, if you’re following along, just follow these and you should be good to go. Let’s do few more examples. So, this is the demo which I’m running. Now, you can see that the first step here is a voice consent gate. It is a safeguard from Hugging Face that only lets a voice be cloned after the speaker explicitly agrees. So, I just generated this consent sentence and then I spoke it like this. I agree to the use of my recorded voice

02:37 for audio generation with the So, once that was done, I verified it and then it said that okay, it is 93% match. And this varies from model to model and then from there it just uh do the speech recognition and once that’s done, I just recorded my own audio and then it has uh given me the cloned voice. Okay, so let me do another one. Okay, now look at this next example.

03:08 This is testing Drama 3’s paired tags which changed the delivery of only the words inside them. For example, look at this. This is a whisper on go on without me, right? And then there is extra emphasis on the most important part and a slower pace on do not skip it, all in my cloned voice and in a single generation. Let me play this. I’m fine. Really. Go on without me. Now, listen. This is the most important part of the video.

03:39 So, do not skip it. How good is that? Really really good one. Okay, next is a scene where I’m testing real human nerves in my cloned voice. So, let me play this first. Tonight is the night and honestly, my hands will not stop shaking. I have been smiling for 12 hours straight and now it is quiet and it is just the two of us. >> [sighs and gasps] >> What if I say something silly? I have never been this nervous in my whole

04:10 life. But when I look at her, everything slows down. Okay, deep breath. I think we are going to be fine. What do you think? Let me know in the comments. Now for this next test, I’m testing if Drama 3 can build real match day excitement. A fast breathless commentary run, a forceful shout on the wicket, a whisper for the tense pause before the last ball, and a laugh of relief at the

04:40 finish. Let’s play this. Last over, 10 runs needed and the whole stadium is on its feet. He runs in, he bowls, it is in the air and oh my goodness, what a catch. That is out. That is absolutely out. Now, it is the final ball. Everyone is holding their breath. He swings and it is a six. We won. I cannot believe it. We actually

05:10 won. That was pretty lame, I would say. Uh let’s do another test. And now I am testing some suspense and fear. So, let’s test it. It was past midnight and the house was completely silent. Then I heard again footsteps upstairs. But I live alone.

05:41 I grabbed my phone and ran for the door but it would not open. It would not open. Who is there? Answer me. And a voice behind me said, “I have been waiting for you.” So, I would say 60/40 percent. 60% it got it right and 40% I think still needs improvement. And now let me check a fast slow goodbye. Well, this is it.

06:13 And that’s it. So, this is, you know, I wanted to show you in real time without editing that this is not a perfect model by any stretch of imagination. It is going to break. You might have to regenerate it. So, be ready for that. And it also supports streaming and as per their model card with the API and some of the coding you can also do the multi-speaker, but I think there is a long, long way to go before this model would be

06:43 suitable for any production use case. Let me also quickly play a couple of demo clips from their hugging face card. And the reason why I am showing you this is just notice a market difference between what I generated in real time in front of you and the curated clip. So, let me play this first one. >> We asked over 20 different people and they all said it was his. >> And look at this point that one. This is the most one uh which I wanted to show you.

07:14 >> you >> [laughter] >> Let me play it from start. >> Ah, [sighs] fine. If we miss it, it’s on you. Well, maybe that’s fair. >> So, you see, um yes, it did a decent job when I generated, but I think this one is way, way better. So, that is a drama three for you. I think the cloned voice sounded quite

07:45 decent to me and the consent step is a thoughtful touch. But as I said, it is not perfect at all. The hardware bar is quite high. The model is also a roughly quite big in terms of download. So check it out. Tags and directions are worth experimenting with if you want to install it locally. But I think results are going to vary from day to day. They also have some continuous editing which I’m I don’t think so is available on this free version. That’s it. Again, if you

08:16 want to help out the channel, please become a member. Thank you for all the support.