Gemma 4 12B: The Encoder-Free Model Explained
Read the Portuguese translation →
Author: Google for Developers · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Gemma 4 12B: The Encoder-Free Model Explained · Channel: Google for Developers · 02:09 · Watch on YouTube
Topics: Data, databases and embeddings, AI models, Multimodal models
Summary
Section titled “Summary”Gemma 4 12B processes audio and image inputs without the larger encoders used by other Gemma 4 models. For audio, it turns segments into amplitude values that can be projected directly to the LLM, reducing the processing required before the model starts working. For images, a small embedder creates embeddings and adds positional information, while the LLM handles giving meaning to the inputs. The video explains that this image embedder is still sizable because projecting image patches requires many parameters.
Key points
Section titled “Key points”- Gemma 4 12B uses no audio or image encoders.
- Other Gemma 4 models use a vision encoder with up to 550 million parameters.
- The conformer audio encoder used by small models has 300 million parameters.
- Encoders process inputs before the LLM starts, increasing time to first token.
- Gemma 4 12B converts audio segments into amplitude values and projects them directly to the LLM.
- For images, a small embedder creates embeddings without extensive encoder processing.
- The image embedder adds positional information so the LLM can identify where image patches belong.
- The image embedder has about 35 million parameters, partly because image patches contain many pixels.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Gemma 4 12B and encoders | The video introduces Gemma 4 12B as a model that processes audio and image inputs without the encoders used by other models in the family. |
| 00:32 | Processing audio without an encoder | The model converts audio segments into amplitude values that can be projected directly to the LLM, avoiding the conformer encoder. |
| 01:06 | A lightweight approach to images | For images, a small embedder creates embeddings and adds positional information so the LLM can interpret image patches. |
| 01:39 | Why the image embedder has 35 million parameters | The embedder’s parameter count reflects the many pixels in image patches, after which the LLM handles giving meaning to the inputs. |
Quotes
Section titled “Quotes”This model is a special one in the Gemma 4 family because it uses no encoders to process the audio and image inputs. — 00:00
This means that the time to first token increases a little bit. — 00:32
And that’s all it takes to remove the encoder for the audio. — 01:06
It’s about 35 million parameters, so quite a bit smaller than the vision encoders. — 01:39
Concepts and entities
Section titled “Concepts and entities”Gemma 4 12B, Gemma 4, encoder, conformer, vision encoder, embedder, LLM, embedding
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 This is Gemma explained in 2 minutes. Let’s get started. This model is a special one in the Gemma 4 family because it uses no encoders to process the audio and image inputs. All other Gemma 4 models, for instance, uses vision encoder, a transformer model to process the image input and give meaning to the embeddings it creates along the way. It’s quite large with up to 550 million parameters. And likewise, these small models use this audio encoder called a conformer.
00:32 It does a similar processing with 300 million parameters, so likewise, quite chunky. Now, what happens with all of these encoders is they first have to process the input, give meaning to the embeddings before the LLM can start working. This means that the time to first token increases a little bit. Instead of using the conformer, the Gemma 4 12 B model simply cuts the audio segments to create sets of amplitude values, and it can project those directly to the LLM because those are all sequential information that the LLM
01:06 knows so well. And that’s all it takes to remove the encoder for the audio. For images, this process is a little bit more involved. And the reason for that is because an image is 3D information. It’s 3D pixels. So, it uses a small embedder to process the input images and just create embeddings out of it. No extensive processing that you see with an encoder. For instance, it needs to add positional information because otherwise, the LLM wouldn’t know which location, which X
01:39 and Y coordinates a patch of an image belongs to. It’s about 35 million parameters, so quite a bit smaller than the vision encoders. The reason why it’s still 35 million parameters is because there are just so many pixels in an image to project. A patch can already be 7,000 pixels. And when you look at the linear projection over here, you have to multiply that with the model’s dimension and that gives a fairly large size. But all the embedder is really doing is creating embeddings with positional information
02:09 and then the LLM can take over handling giving meaning of the image and audio inputs. Now, if you want to learn more about the encoder free Gemini 4 model, here are a couple of cool and interesting resources you can check out and I’ll see you in the next video.