Embedding Gemma 2: Multimodal Search & RAG (Free Colab)

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Embedding Gemma 2: Multimodal Search & RAG (Free Colab) · Channel: Prompt Engineering · 18:50 · Watch on YouTube

Topics: Data, databases and embeddings, AI developer tools, Local and open-source models, RAG, search and information retrieval

Summary

The video introduces Embedding Gemma 2, a compact open-weight model that places text, code, images, video, and audio in a shared vector space for multimodal search and other tasks. It explains how the model processes different modalities, its shared embedding space, modular loading, and Matryoshka dimensions, then demonstrates its use in a free Colab notebook. In the creator’s tests, text-to-photo retrieval performs well, voice queries can retrieve matching images, and video and document search work without captions or text extraction. Audio retrieval is less reliable for environmental sounds, so the video recommends testing it on your own data.

Key points

The model maps text, code, images, video, and audio into a shared vector space, enabling cross-modal retrieval.
Unlike pipelines that caption images or transcribe audio first, direct multimodal embeddings can retrieve information not included in a caption or transcript.
Vision and audio encoders convert their inputs into tokens that are processed alongside text by a shared backbone.
The model supports mixed inputs and produces a 768-dimensional vector; its input has an 8,192-token context window.
Matryoshka representation learning allows embeddings to be truncated to fewer dimensions, reducing storage but potentially lowering multimodal retrieval performance.
The notebook’s photo search returned the correct image first about 86% of the time and placed it in the top five 97% of the time across 5,000 caption queries.
Spoken queries can retrieve images without a speech-to-text step, while text queries can also search video moments and scanned document pages.
In the sound-effects test, the correct clip ranked first 24% of the time; the presenter advises testing environmental audio on your own data.
A full-model index can be searched using the text-only model, allowing a smaller model to run searches over multimodal data.

Chapters

Start	Chapter	Summary
00:00	Introduction and multimodal search demo	The video introduces the model and demonstrates searching for photos with a spoken query.
01:03	Why multimodal retrieval matters	The presenter explains the limits of text-only retrieval and reviews earlier approaches to searching across modalities.
04:16	How the model works	The video describes the model’s encoders, shared backbone, training approach, modular loading, and variable embedding dimensions.
09:08	Colab setup and photo retrieval	The notebook setup is covered, followed by tests of cross-modal similarity and text-based photo search.
13:27	Voice and sound retrieval	The presenter tests spoken queries against photos and evaluates the model’s weaker performance on environmental sounds.
15:09	Video and document search	The notebook searches video moments and document pages directly from their visual content.
16:45	Embedding size, model loading, and conclusions	The presenter compares retrieval at smaller embedding dimensions, demonstrates searching with the text-only model, and summarizes the model’s strengths and limitations.

Quotes

You don’t need speech to text transcription. — 00:33

You can take a product listing with description, a photo, and a short video, and embed all of it into one vector — 05:48

the right photo comes back first about 86% of the time — 12:21

the audio encoder is strongest on speech — 14:34

you can build the index with the full model and then only use the text part of the model to do search over images, videos, or audios. — 17:48

Concepts and entities

Embedding Gemma 2, Gemini Embedding 2, Colab, Sentence Transformers, CLIP, ImageBind, ColPali, Matryoshka Representation Learning, mean pooling, Flickr, ESC-50, Big Buck Bunny

Full transcript (original language)

00:00 Google just released embedding Gemini 2. And this is a big news for people who are interested in multimodal retrieval augmented generation. It’s a small open embedding model that puts text, code, images, videos, and audios into the same vector space. And it’s small enough that you can actually run this on a phone. So, it’s going to open up so many different possibilities for developers. The best part is that you can use this beyond retrieval. You can do classification and clustering on different modalities using the exact

00:33 same model. So, in this video, I want to show you different capabilities that this new embedding Gemini 2 offers. But before I show you how it works, let me show you a quick demo. >> A dog jumping into a lake. >> This demo shows cross modality search. The speech goes into the model, and it will retrieve four different pictures. And these are the same four photos I get if I type the exact same sentence. You don’t need speech to text transcription.

01:03 Okay. So, in this video, I want to cover why multimodal retrieval matters, how people solved it before this, how this model works under the hood, and then we will walk through a Colab notebook that you can run yourself on a free GPU. All right. But let’s talk about why you would even need something like this. Most of the rag or retrieval augmented generation systems we build today are based specifically on text search. You take your documents, split them into

01:34 chunks, turn each chunk into an embedding, and when a question comes in, you find the chunks that are closest to it in that embedding space. But think about where most of our data actually lives. These are things like photos on your phone, screen recordings, voice memos, meeting notes, slide decks, scanned documents. A text-only retrieval cannot see it in any way. Say, you want to find a clip where your dog catches a frisbee and there’s no text in the video itself.

02:04 A text-based retrieval will not be able to find that specific video for you. Now, the same applies to agents because agents’ memory is only as good as what it is able to do retrieve. So, multimodal retrieval is very critical for search-based applications. Now, let’s talk about how people have been solving this. The most common approach is to turn everything into text first. You run a captioning model on every image, a speech-to-text model on every audio file, and then embed the text. It works,

02:38 but now you have three models chained together. And the bigger problem is that you can only find what the caption or transcription wrote down. For example, if the captioner said “a dog in the park” and never mentioned the frisbee, your search is not going to return that clip. Now, Google’s paper on Gemini Embedding 2, which is their larger API model, measured this specific phenomenon for audio. Embedding the audio directly scored 73.99 on their sound retrieval benchmark

03:12 versus 70.40 when they embedded the transcript instead. Now, the second approach came from CLIP in 2021. You train two separate models, one for images and one for text. So, that photo and its caption is going to end up very close to each other. Now, this approach made text-to-image search really good, but it only covers one pair of modalities and it’s trained on short captions.

03:42 Then, in 2023, Meta’s ImageBind extended the idea of six modalities by aligning each one to images. And for documents, Call Polly started embedding screenshots of pages directly, which skips OCR, but it stores many vectors per page. So, the index gets really big really quickly. I have quite a few videos on that if you’re interested. Links to those are going to be in the description. The newest approach is one model with a single backbone that reads every modality. Google’s Gemini embedding 2

04:16 works this way, but it’s an API model. And this is where embedding Gemini 2 comes in. It brings the same design down to under a billion parameters with open weights. Okay, so here’s how it actually works. First, a quick refresher. An embedding is basically a list of numbers that captures the meaning of an input. Inputs that mean similar things get similar numbers. Uh so, you can compare meaning instead of exact words.

04:47 Now, a model that reads text cannot read pixels or sound waves directly. So, embedding Gemini 2 has two small encoders in front of it. The vision encoder, which is 170 million parameters, takes an image or a video frame and turns it into tokens. The audio encoder, which is 300 million parameters, does the same thing for audio. Those tokens then go into the same sequence as the text tokens.

05:17 So, from the model’s point of view, a photo is just a few hundred uh more tokens in the input. By default, an image costs 280 tokens. A video frame costs 140, and a second of audio costs about 25 tokens. The context window is 8,192 tokens. So, one input can hold about 29 images, about 58 video frames, or around 5 and 1/2 minutes of audio.

05:48 And that’s also how you get mixed input. You can take a product listing with description, a photo, and a short video, and embed all of it into one vector, which is extremely neat. Now, the beauty is that all of this goes through one shared backbone, which is 270 million parameter text model based on Gemma 4. At the end, the model averages the output of every token into one vector, which is called mean pooling, and then projects it into those 768 numbers or dimensions.

06:22 Now, let’s understand how does a photo of a flamingo end up next to the sentence about flamingo? So, Google hasn’t published a technical report on this model yet, but they say it’s built from the same technology as the Gemini embedding. The paper for Gemini embedding models describes contrastive training. You show the model pairs that belong together, like a photo and its caption, or a clip and its transcript, and you train it to pull those pairs closer and push everything else in the batch away.

06:57 Now, if you repeat this process over a huge mixed data set, you are going to see that every modality ends up in one shared map. Okay. Now, there are two more design choices that matters when you want to deploy this into production. The first one is Matryoshka representation learning. The model is trained so that the first 512 to 256 or even 128 of those numbers or dimensions still work as an embedding on their own. So, essentially, you can cut those dimension into smaller dimensions, and

07:30 they will still work. Smaller vectors mean a smaller index. So, at 128 numbers, you store six times less, but there is a catch. In Google’s own table, the text barely changes at 128 dimensions, while the multimodal score drops from 59 to about 46. And I’ll show you some example of this in the notebook. Okay, the second is modular loading. The vision and audio encoders are optional. If you only have text and code, you load

08:03 those 270 million parameters. If you add vision, that’s 420 million. And if you add audio instead, that is going to be 570 million parameters. The full model is 740 million. Now, all of these share the same vector space, so you can index your photo once at the full model and search them from a phone with a small one. So, with quantization, Google says that the text-only version needs about 191

08:37 MB of RAM on a Pixel 7 Pro. And the full model is going to be about 567 MB. Okay, so the beauty is that it’s actually much better compared to the previous embedding Gemma model, and also has pretty powerful retrieval capabilities compared to some of the bigger models in the same space. I’ll put some links to the benchmarks in the video description if you are interested. Okay, so that’s enough theory. Let’s open the notebook and see

09:08 how well this works in practice. Including some places where it did not work that well. Okay, so here’s a notebook. Link is going to be in the description. And it runs on free T4 GPU in Colab in about 15 minutes. So, first let’s talk about the setup. The first cell installs sentence transformers with image and videos extras. And there are two gotchas here that I ran into. The video extra doesn’t

09:39 install torch codec, and without it video decoding fails. So, I had to add that myself, and I skipped the audio extra because it pulls in an older version of NumPy that pip compiles from source on Colab for several minutes. Audio actually works fine without it. So, keep that in mind if you run into some issues here. Then, reload the model, and this part is actually important. Don’t run this model in float 16. The model card says it’s

10:10 activation overflow, and instead of an error, you get NaN or wrong embeddings. A default doesn’t support floating point 16 natively. So, here we use the float 32. On a newer GPU, use Bfloat 16, and that will work. For data, I’m using 1,000 Flickr photos that each come with five human handwritten captions, 50 everyday sound clips from a data set called ESC-50, and then uh first 5 minute of the open

10:40 movie Big Buck Bunny, and uh the activation is all you need paper as a PDF. So, these are different modalities that we’re going to be testing on. Okay, so let’s start with the core idea from the first part. I took six concepts like a dog, a crying baby, and ocean waves, and each one exists as a sentence, a photo, and a sound. If everything really lives in one space as it’s described, the matching pairs should score highest, which shows up as

11:11 a bright diagonal here. So, text to photo is a clean diagonal. Text to sound gets most of it right. And a photo to sound is the noisiest, which kind of makes sense because it compares a picture directly with a sound clip with no text in between. Now, keep that in mind because sound comes back later with some really interesting results. Also, notice that all these scores sit between about 0.5 and 0.7. So, use them to rank results and don’t pick a fixed cutoff unless you have tested this on your own data.

11:46 Picking the threshold can really impact on the results that you’re going to get. Okay, let’s talk about photo search. So, now let’s build a real index. Every photo gets embedded once, which is the slow part. This takes about 8 minutes on T4. After that, every search is one text embedding and one comparison. So, remember the example from the start, a dog catches a Frisbee. Here it is, the same example. The top two photos both have Frisbee in the air. But a few good examples don’t tell you much. So, we can actually measure it.

12:21 So, every photo here has five captions written by people, which gives us 5,000 queries where we know the right answer. Now, in my experiments, the right photo comes back first about 86% of the time out of the 1,000 photos, and it’s in the top five 97% of the time, which is actually pretty impressive for the model of this size. Especially when it comes to retrieval, you don’t want to use the top one, but normally people use top K. K can be five

12:53 or 10. And for top five or K5, this actually is pretty good. Now, here is the best part. The text side understands over 100 languages because Gemma is multilingual. So, a German query for a dog running in the snow returns dog in the snow. In Spanish, I asked for a woman playing the guitar, and the first two are right. The third one is a woman playing a harp. So, it is close, but not quite. Now, the beauty is that the query doesn’t have to

13:27 be text either. If I use the surfer photo as the query, I get two more surfing shots and then people wakeboarding. So, it matches people riding boards on water, not just the color blue. Which is pretty amazing. You can actually build private photo-based search things on your own gallery. Which is pretty incredible. Okay. So, next let’s look at voice search. So, this is the demo from the start of the video. I generate a spoken question, a

13:59 dog jumping into a lake. And pass the audio right to the model. With no transcription step. Now, the four photos it return were the same photo if I write the same sentence in text. And the same happens for a little girl on a swing, right? So, it basically is doing this cross-modality search in the same space. Okay. Now, here’s where it gets weaker. So, I took 50 everyday sounds, one clip each, and for every sound name, I check whether its clips comes back first. That

14:34 happens actually 24% of the times. Now, random guessing would get 2%. So, the model is picking up something, but that’s the long way from 86% that we saw for photos, especially for K or pass at one. You can see it in the queries. Someone laughing finds the laughing clip. And a toilet flushing finds the toilet. But a dog barking comes back as a crow and a rain falling comes back as wind. So, the audio encoder is strongest on speech,

15:09 which is what the voice demo showed us as well. If you need sound effects or environmental sounds, I think you’ll need to test it on your own clips first. Okay. Now, let’s look at the video. So, I cut the first 5 minutes of um Big Buck Bunny into five uh second chunks and embed each one in as a video. By default, the model looks at one frame per second and a text query that returns the timestamp. So, a bird stretching its wing on a branch lands at 15 seconds for

15:42 our data set. And a dark hole in the ground is the rabbit’s burrow at 30 seconds. And the rabbit smelling a white flower is at a minute and five. And the purple butterfly is at 3 minutes 20 seconds. So, finding a moment in the video works without any captions, which is incredible. Uh and keep in mind, it’s exactly the same model under the billion parameter, which works across different modalities. Okay. Now, let’s look at documents. So, I turned every page of the attention

16:15 paper into an image and embedded the picture of the page with no text extraction at all. Now, asking for the diagram of the Transformer architecture returns page three, which is this very famous figure, which started basically everything. Multi-head attention returns page four, and the table of blue score returns page eight. So, I tried giving each page four times more image tokens, which is a setting that you can change. So, it took about nine times longer and

16:45 found the same pages. So, keep the default unless you need fine details like small numbers in a dense table. Okay. Now, uh remember the catch with the smaller factors because it does support Matryoshka’s embedding. So, here it is on our data. The right photo comes first 86% of the time. At 256, it’s 84 with three times less storage. Uh but at 128, it drops to 75.

17:16 So, you will actually need to play around with the final dimensions that you want to use because at um certain point, the accuracy drop is going to be pretty significant. In our code, you pass truncate dimension and normalize embeddings to encode and define those values. Okay, in this last section, I’ll show you uh how do you load things that you need and you drop things that you don’t. So, here I load only the text part of the model, which is 270 million parameters and uh search the photo index

17:48 we built with the full model. Remember, you can build the index with the full model and then only use the text part of the model to do search over images, videos, or audios. The query embedding comes out exactly the same and so does the recall. So, you can index your photo once on a server and search them uh from a phone uh with the small model. So, you can essentially [snorts] run search over photos, videos with a wise note without turning anything into text, right? So, for photos, videos, and

18:20 documents, yes, and it works really well for a model this size. For sound effects, as I said, it’s not there yet. The beauty is that you can fine-tune this model for your own applications. By the way, let me know if you’re interested, I’ll make a follow-up video on how to fine-tune the embedding Gemma 2 model. And if you’re building um search or rag over that kind of data, this is really the first open model that I would try. And towards the end of the notebook, I have a few rules that you can try to get

18:50 the retrieval right. Anyways, this is I think really exciting, especially the embedding model coming out of Google and it’s multimodal. You can run it on a phone. It can power projects like local GPT. I’ll probably include that in the project itself. But anyways, do let me know if you found this interesting, I’ll make some follow-up videos. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one. >> [clears throat]