ByteDance Just Made a 33B Video + Audio Model Run in 4 Steps (DMAD)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: ByteDance Just Made a 33B Video + Audio Model Run in 4 Steps (DMAD) · Channel: Prompt Engineer 48 · 05:12 · Watch on YouTube

Topics: Model inference and optimization, AI models, Local and open-source models, Multimodal models

Summary

The video explains DMAD, a ByteDance and Texas A&M research method for distilling a video-and-audio model so it can generate clips in four steps instead of the teacher model’s 50. Rather than using DMD’s auxiliary diffusion model as a critic, DMAD uses a discriminator whose logit gradient provides the teaching signal, while cached teacher samples reduce training costs. Its two-headed design uses both teacher samples and real videos, with gap-based re-weighting to improve the training signal. The video presents reported speed and quality results, then notes that trying the model requires substantial GPU memory and accepting the base model’s license.

Key points

DMAD is presented as a method for making a 33-billion-parameter audio-and-video model generate in four steps.
The teacher model, MiniMax H3, uses 50 denoising steps per clip.
DMD uses an auxiliary diffusion model as a critic; DMAD replaces it with a real-or-fake discriminator.
The authors say the trained discriminator’s logit gradient provides the signal that DMD estimates with its second model.
DMAD uses one discriminator update per student update, compared with five critic updates per student update in DMD 2.
Teacher samples are generated once and cached, so the teacher does not need to generate them live during training.
Two heads share a backbone: one compares the apprentice with teacher samples, and the other with real videos.
Gap-based re-weighting gives the teacher more weight at noise levels where its samples are closer to real data.
The reported benchmarks include faster training and, in some cases, results that surpass the teacher model.
The video says inference requires a 180 GB GPU, and the base model download is about 170 GB.

Chapters

Start	Chapter	Summary
00:00	The four-step generation challenge	The video introduces DMAD and explains the speed-versus-quality challenge of reducing a 50-step video model to four steps.
01:34	Replacing the critic with a discriminator	DMAD uses a real-or-fake discriminator to provide the teaching signal, while cached teacher samples reduce training overhead.
03:06	Re-weighting and reported results	The video explains gap-based re-weighting and reviews reported training-speed, benchmark, and human-study results.
04:41	Requirements and closing notes	The presenter outlines the GPU and download requirements, clarifies they have not run the model, and points viewers to the paper and repository.

Quotes

a way to make a 33 billion parameter audio and video model generate in just four steps. — 00:00

Is this painting from the apprentice or from the real thing? — 01:34

Same teaching signal, no second model. — 02:04

So no this is not running on my 8 gigabyte laptop. — 04:41

Concepts and entities

DMAD, DMD, DMD 2, MiniMax H3, ByteDance, Texas A&M, H200, LoRA

Full transcript (original language)

00:00 All right, in this video we’ll see D-Mat, a way to make a 33 billion parameter audio and video model generate in just four steps. And you can see that the clips come out with real sound as well. Now, mind you, this is a research release from ByteDance and Texas [music] A&M, and the code and the weights are open. Let’s go ahead and see how it works. Now, the problem is these video models are slow. The teacher model here, MiniMax H3, takes 50 denoising steps for every single clip. So, you type a prompt and then you wait four steps instead of 50. Sounds really great, but the catch

00:31 is the quality usually falls apart. So, here’s how I’m going to explain this. Think of a master painter. He paints a picture in 50 careful brushstrokes. Now, we want an apprentice who can paint the same picture in just four strokes. The master is the teacher, the apprentice is the student. The question is, how do you train the apprentice? The popular answer is something called DMD, distribution matching distillation. What it does is it hires a second person, a critic, who watches everything the apprentice paints and keeps writing notes on how his style differs from the master’s in AI terms.

01:03 That critic is an auxiliary diffusion model, a whole second model trained alongside the student. Now, the trouble is the apprentice keeps changing, so the critic is always a little out of date. And if the critic is out of date, his notes are wrong, and training becomes unstable. The fix in DMD 2 was to let the critic update five times for every single update of the student. And on top of that, the master has to paint live during training, too. So, you pay for all of it in memory and in time. This is exactly what D-Mat fixes. Instead of a critic writing detailed notes, you hire

01:34 a judge who answers one question: Is this painting from the apprentice or from the real thing? That is it, real or fake? Now, if we go a bit deeper, here is how it works in technical terms. A discriminator looks at a noisy video and outputs a single number, a logit. The authors prove that when this discriminator is trained well, the gradient of that logit is exactly the signal DMD was estimating with its second model. The paper calls it the log density ratio. Too technical for this video, but basically the judges score

02:04 already contains the critics notes. Same teaching signal, no second model. That means just one discriminator update for every student update, not five. And since the master only needs to supply paintings, they generate the teacher samples once, cache them, and the master never has to paint life during training again. Now, you might be thinking if the apprentice only copies the master, the best he can ever be is as good as the master. Well, actually, that is why DMAD has two heads on one shared backbone. One head compares the apprentice against

02:35 the teacher samples, the other compares him against real videos. So, the apprentice can actually end up better than the master. And you can see that in the ablations. If you remove the real data head, the four-step FID on COCO goes from 14.47 up to 21.37. And if you remove the teacher, the semantic score on the video benchmark drops from 79.55 to 75.36. There is one more trick called gap-based re-weighting. The real data head also scores the teacher samples, so it knows

03:06 how far the teacher is from real at each noise level. Where the gap is small, the teacher is already close to real, so DMAD gives the teacher more weight there. Turn this off and the video benchmark total drops by 1.14. Now, let’s look at the numbers because this is where it gets good. One generator update on the 33 million minimax model, including the critic updates for DMD 2, takes 818 seconds with DMAD 233 seconds. On one 2.14 billion, it goes from 162.9 seconds down to 38.5, and on ImageNet

03:38 2.321 seconds down to 0.496. That is on H200 GPUs. On quality, with a single step on ImageNet 64, DMAD reaches an FID of 1.04 with a projected discriminator. On SDXL with four steps, 14.47. On COCO, which beats the SDXL teacher itself and for video four step one 14B hits a V bench total of 85.15 while the 50 step teacher gets 83.59. For Minimax H3 they ran a human study.

04:09 387 prompts three annotators each people preferred D Mad over DMD 279.1% of the time and over RCM 84.6% with ties excluded. And mind you there were quite a few ties a third of comparisons against DMD 2. Get this the training itself was only 800 iterations on eight H200s using a rank 128 Laura. Now if you want to try it the repo has inference.py four steps 1344 by 768 124 frames about 5.2 seconds at 24 frames per second with

04:41 stereo audio. But mind you you need 180 gigabyte GPU and the base model download is about 170 gigabytes. So no this is not running on my 8 gigabyte laptop. I have not run it myself in this video so everything here comes from the paper and the repo. You also have to accept the Minimax H3 community license on hugging face first. Still the idea is really good and the weights are only 1.4 GB gigabytes each. So in this video we have seen how D Mad swaps the critic for a simple real or fake judge. Go ahead and

05:12 check out the paper and the repo and let me know how it goes. I will see you in another one.