NVIDIA Just Deleted the VAE and the Vision Encoder (PixelUMM Explained)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: NVIDIA Just Deleted the VAE and the Vision Encoder (PixelUMM Explained) · Channel: Prompt Engineer 48 · 05:11 · Watch on YouTube

Topics: AI evaluation and benchmarks, AI content generation, AI models, Multimodal models

Summary

PixelUmm is presented as a visual model that processes raw image and video pixels without a VAE or vision encoder. It uses patch or tube tokens and a decoder-only transformer with separate understanding and generation experts that share self-attention. The video describes its design choices, sample outputs, benchmark results, and limitations, including occasional visual errors and a non-commercial research license. The creator argues that direct pixel processing could simplify visual-model pipelines, while noting that larger-scale improvements remain a possibility rather than a certainty.

Key points

PixelUmm is described as generating and understanding images and video directly from pixels, without a VAE or vision encoder.
Images are divided into 16-by-16 patches, each converted into a token by a linear layer; video uses tubes of four frames by 16-by-16 pixels.
The model is a decoder-only transformer based on Queen 3 8 B, with 36 layers and about 15.2 billion parameters.
Understanding and generation use separate experts but share self-attention.
A linear output head predicts clean pixels directly over 35 denoising steps, without a latent space.
The paper’s tests favor 16-by-16 image patches over 32-by-32 patches; a convolutional pixel-shuffle head reduces visible grid artifacts but costs more compute.
The video reports competitive but uneven understanding benchmarks: PixelUmm scores 90.4 on DocVQA, 83 on ChartQA, 94.3 on CountBench, and 41.7 on MMU.
Reported limitations include merged overlapping figures, inaccurate counts or fingers, and physically implausible motion.
The checkpoint is under a non-commercial research license, and the repository lists training requirements of seven or more GPUs with 48 GB each.

Chapters

Start	Chapter	Summary
00:00	PixelUmm and the usual visual-model components	The video introduces PixelUmm and explains how VAEs compress and reconstruct images while vision encoders convert images into features for understanding.
01:32	A model that works directly with pixels	PixelUmm turns image patches and video tubes into tokens, then uses a transformer with shared attention for understanding and generation.
02:34	Outputs and design findings	The video shows generated examples and reviews findings about patch size, output heads, and training in pixel space.
04:09	Benchmarks, limitations, and what may come next	The creator weighs benchmark results against known errors and licensing and hardware requirements, then discusses possible implications of pixel-based modeling.

Quotes

One model, raw pixels, images and video understanding and generation. — 00:00

PixelUmm’s answer is simple, skip both and just feed the pixels in. — 01:32

Pixels in, pixels out. — 02:04

The point is that one encoder-free model gets this close. — 04:09

that’s Pixel UMM, no VAE, no vision encoder, just pixels. — 05:11

Concepts and entities

PixelUMM, NVIDIA, VAE, ViT, Qwen3 8B, GitHub, Hugging Face

Full transcript (original language)

00:00 All right, Nvidia just released PixelUmm and it throws away the VAE and the vision encoder completely. One model, raw pixels, images and video understanding and generation. Let’s break it down. Now, before the theory, look at this. Everything you see on screen was generated by this one model straight from a text prompt, 4 seconds, 24 frames per second. And mind you, there is no VAE decoding any of this. So, to understand why this is a big deal, we need two things that almost every image and video model depends on,

00:30 a VAE and a ViT. Let’s start from the basics. A VAE, a variational autoencoder, is basically a compressor. A video is huge, millions of pixels per frame, so the model first squeezes it down into a small hidden grid called latents, does all its work there, and at the end a decoder blows the latents back up into pixels. So, you can see that the VAE is a separate model trained separately sitting at the front and the back. The catch is whatever the VAE can represent, the generator can never

01:01 create fine textures, small text, sharp edges. The VAE becomes a ceiling for quality, and it’s one more thing to train, store, and keep in sync. Now, the ViT, a vision transformer, this one is on the understanding side. When you ask a model what is in this picture, the image goes through a vision encoder first. It cuts the picture into patches and turns them into features that the language model can read. Again, a separate pretrained model with its own limits. So, a typical unified model today has a ViT to understand, a VAE to

01:32 generate, and a language model in the middle gluing them together. Three parts, three sets of compromises. PixelUmm’s answer is simple, skip both and just feed the pixels in. An image is cut into 16 by 16 patches. Each patch is 768 raw numbers, and one single linear layer turns it into a token. For video, the unit is a tube, four frames by 16 by 16, which is 3,072 values. One linear layer, one token. That’s the entire front end. Then those tokens go into a decoder-only transformer built on Queen

02:04 3 8 B, 36 layers. Now, here’s the nice part. If you look at the diagram, there are two experts. The understanding expert handles text and clean pixels, and the generation expert handles the noisy pixels, but they share the same self-attention. So, everything talks to everything in one sequence. The total is about 15.2 billion parameters. For understanding, the language head writes text like describing a video for generation, a linear head predicts clean pixels directly denoised over 35 steps, no latent space anywhere. Pixels in,

02:34 pixels out. And here’s what comes out. A fox at dawn, a golden retriever in the wind, an otter surfing, and neon Tokyo street. All of these are 832 by 464 96 frames from just a text prompt. Now, if you want to go deeper, the paper has some really interesting findings. Let’s go through a few. First, patch size. They tested 16 by 16 patches against 32 by 32, and 16 keeps a lower training loss. The takeaway is that squeezing the image harder makes generation harder to learn. For video, they settle on 16

03:05 spatial by four temporal. Second, the output head. A plain linear head has a problem. At high guidance, you can actually see the 16 pixel grid in smooth areas like a sky or a wall. They tried convolutional heads with pixel shuffle that fixed this, and the best balance costs about 9.6 times more compute in that head. Interesting thing, the released checkpoints still use the simple linear head because switching needs more training. Third, pixel space against VAE space. Training directly on pixels works, and the gradient norms are

03:35 nearly the same as with the frozen 1 2.2 VAE, though pixel space does show some occasional loss spikes. And the 8 billion model reaches the same loss in roughly a third of the training steps of the 1.7 billion one. Now, let’s be honest about the numbers. On image understanding, it’s competitive, not the leader. DocVQA 90.4, ChartQA 83. On count bench, it scores 94.3, ahead of the baselines in their table. But on harder ones like MMU, it gets 41.7, where Qwen-VL gets 69.6. So, the point

04:09 isn’t winning every benchmark. The point is that one encoder-free model gets this close. There are limits, too. When several animals or people overlap, their bodies can merge or the count drifts. Hands can have the wrong number of fingers. Motion can be physically implausible. And the checkpoint is under a non-commercial research license. So, this is for research, not for shipping a product yet. If you want to try it, the code is on GitHub and the checkpoint is on Hugging Face. Inference is one command with tasks for text-to-image, text-to-video, image understanding, and

04:40 video understanding. Mind you, the repo lists training on seven or more GPUs with 48 GB each. So, check the requirements before you try it on your own system. So, what’s the future? This is my take, not the paper’s. If raw pixels work, the VAE stops being a quality ceiling, and the pipeline gets simpler. One model, one training loop. Add the scaling result, the bigger model learning faster, and it suggests larger versions should keep improving. And with image-to-video and video editing already in the same model, this could become one general visual model. We’ll see. So,

05:11 that’s Pixel UMM, no VAE, no vision encoder, just pixels. Links are in the description. Go ahead and try it out, and let me know how it goes. I’ll see you in another one.