Per-Layer Embeddings (PLE) in Gemma 4 explained
Read the Portuguese translation →
Author: Google for Developers · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Per-Layer Embeddings (PLE) in Gemma 4 explained · Channel: Google for Developers · 01:37 · Watch on YouTube
Topics: Data, databases and embeddings, AI models, Local and open-source models
Summary
Section titled “Summary”Gemma E2B and E4B use per-layer embeddings (PLE), with the “E” referring to “effective.” PLE expands a token’s representation so it can have a different embedding at each layer, providing the model with more information. Although the embedding table is large, inference uses only the embeddings for the tokens encountered. The video describes this as adding representational power without increasing the parameters used during computation, since the embedding parameters are stored on flash storage.
Key points
Section titled “Key points”- Gemma E2B and E4B use per-layer embeddings.
- The “E” in E2B and E4B stands for “effective.”
- A token embedding layer gives an input token a numerical representation.
- PLE expands a token ID into a three-dimensional representation.
- A token can have different embeddings at different layers.
- During inference, only a small portion of the large embedding table is used.
- The architecture and encoders are described as the effective parameters used for computation.
- Per-layer embeddings provide more representational power without increasing parameters used during computation; the embeddings are stored on flash storage.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Token embeddings and PLE | The video introduces PLE in Gemma E2B and E4B and explains how it builds on token embeddings. |
| 00:31 | Layer-specific representations | PLE gives a token different embeddings at different layers, while inference uses only a small portion of the embedding table. |
| 01:02 | Effective parameters and computation | The video explains which parameters are used in computation and how layer-specific embeddings add representational power without increasing computation parameters. |
Quotes
Section titled “Quotes”It stands for effective and learns from the token embedding layers. — 00:00
the token hi has a different embedding on layer one than it does on for instance layer six — 00:31
They are inserted at each given layer to provide the model with more information. — 01:02
giving it more representational power. — 01:37
Concepts and entities
Section titled “Concepts and entities”Per-Layer Embeddings (PLE), Gemma 4, E2B, E4B, token embedding layer, effective parameters
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 This is per layer embeddings explained in 2 minutes. Let’s get started. Gemma for E2B and E4B use per layer embeddings and the E in E2B references that. It stands for effective and learns from the token embedding layers. The token embedding layer, when it receives an input like the token hi, it tries to extract an embedding to give it a numerical representation. And PLE takes that ID and expands upon it. It is a three-dimensional representation of
00:31 these token embedding layers. And it means that the token hi has a different embedding on layer one than it does on for instance layer six to give it a more diverse representation. So when we see the token hi, we only need those embeddings. And even though this is a very large embedding table, we use a very small portion of it during inference. This means that the architecture and the encoders, those are really part of the
01:02 computation, but the per layer embeddings and the token embedding table, you only need a very small portion for of that. Because when we see the token hi, you only extract those specific embeddings. And that’s why we call the architecture and the encoders the effective parameters because those are the only parameters actually used for computation. They are inserted at each given layer to provide the model with more information. So whereas the token embedding layer has one representation for the token hi, a per layer embeddings adds layer
01:37 specific embeddings for that specific token, giving it more representational power. And it’s a very nice way to increase the performance of the model without actually increasing parameters because these parameters are stored on flash storage. You don’t need them during computation. Now, if you want to learn more about per layer embeddings, here a couple of resources that you can use and I see you in the next video.