Training Your Own Embedding Model Is Not As Hard As You Think

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Training Your Own Embedding Model Is Not As Hard As You Think · Channel: Prompt Engineering · 12:33 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, AI models, RAG, search and information retrieval

Summary

The video explains how to fine-tune an embedding model using question–passage or label–audio pairs, with LoRA adapters and in-batch negatives. It emphasizes avoiding duplicate items within batches, matching training prompts to production prompts, and evaluating on data split by source. In two experiments, fine-tuning improved retrieval for the creator’s sounds and video transcripts, but did not teach general skills for unseen sound labels and reduced general text search performance by about four points. The video concludes that fine-tuning can be practical, provided you test both the target task and other capabilities that may be affected.

Key points

Embedding models learn relationships between items rather than producing a specific text answer, so training data can consist of matching pairs.
In-batch negatives let the model treat other passages in a batch as incorrect matches without separately labeling negative examples.
A no-duplicate batch sampler helps prevent a correct passage from being treated as a negative for another matching question.
Training and production should use the same prompt and document format.
LoRA trains small adapter matrices while keeping the original weights frozen, rather than updating the full model.
Evaluation should use a held-out test set split by source, such as article, video, or recording.
On ESC-50 sounds, top-result accuracy on unseen clips improved from about 25% to about two-thirds, but performance did not improve on sound labels withheld from training.
Fine-tuning on video transcript passages improved retrieval from about two-thirds to three-quarters after less than three minutes of training.
Photo and voice search did not change in the sound experiment, while general text search dropped by about four points.
The speaker recommends checking model behavior after saving and reloading an audio adapter, because the saved processor caused audio clips to be truncated in one test.

Chapters

Start	Chapter	Summary
00:00	What embedding fine-tuning does	The speaker introduces a fine-tuned video-search demo and explains how embedding training uses matching pairs and in-batch negatives.
03:09	Batching, prompts, and LoRA	The video covers duplicate-pair pitfalls, prompt consistency, LoRA adapters, and how to evaluate whether the model generalizes.
06:19	Fine-tuning for sound search	Using ESC-50, the speaker trains on audio-label pairs and reports improved retrieval on clips, alongside tests of unseen labels and other model capabilities.
09:58	Fine-tuning for transcript search	The speaker describes creating question–passage pairs from video transcripts and reports the retrieval results and text-search trade-off.
11:30	Takeaways	The speaker summarizes the benefits and limitations of fine-tuning and recommends using pairs, source-based test splits, and post-training checks.

Quotes

the wrong answers for free without labeling a single one. — 03:09

The prompt template that you use for training should be exactly the same as the one that you’re using in production or inference. — 04:12

The model learns the labels you give it, not hearing in general. — 08:54

A few minutes of fine-tuning made it clearly better on my own sounds and my own videos — 11:30

Concepts and entities

Embedding Gemma 2, LoRA, multiple negative ranking loss, no duplicate sampler, Sentence Transformers, CLIP, Unsloth, ESC-50, Gemini 3.8 Flash

Full transcript (original language)

00:00 Embedding Gemini 2 is great, open model for multimodal retrieval, but what makes it even better is that you can fine-tune it on your own data. Now, this matters because an embedding model off the shelf only knows the general data it is trained on, but it has no clue about your documents, your product names, or sounds your applications care about. Now, before I show you how it works, let me show you a quick demo. This is a version of Embedding Gemini 2 that I fine-tuned on Transcribe for my own

00:31 YouTube channel. I type a question and it returns the exact moment in a video where I talk about it with a timestamp that I can click. So, the question I want to answer in this video is whether a few minutes of fine-tuning can make this model noticeably better at your own data without breaking what it already does well. In this video, I want to show you how fine-tuning an embedding model works, and then we will walk through a notebook where I fine-tuned it on two of my own

01:02 examples. Okay, so let’s talk about what makes fine-tuning an embedding model different from fine-tuning a large language model. When you fine-tune an LLM, you show it an input and exact text you want it to write back. So, training data already contains the right answer word for word. Now, an embedding model does not write anything. It turns an input into a list of numbers, and those numbers are only useful in relationship to each other.

01:34 This is what people usually call the embedding space. So, there is no correct list of numbers or embeddings that you could write down as a target. The only thing you can tell the model is which things should end up close to each other. Now, this changes what your training data looks like. Say you run support for a smart home company. And you want a search box that answers customer questions from your help center. Each training example is a pair, which

02:06 is a customer question and a passage that answers it. You don’t need scores or label wrong answers. Only questions and passages that belongs to each other. But pulling each question towards its passage is not enough on its own because the model could put everything in the same spot and still score well. So, let’s talk about how the model learns what to push away and what to bring closer. During training, the model sees a batch of pairs at once. For

02:38 example, a batch of 32 pairs. For each question, its own passage is the right answer and every other passage in the batch is treated as the wrong answer. So, you don’t actually have to collect negative examples. You automatically are going to have access to negative examples if you collect these pairs. So, the loss pulls each question towards its own passage and push it it away from all the others. Which means you get the

03:09 wrong answers for free without labeling a single one. So, the easiest mental model is a multiple choice question where other pairs in the batch are the wrong options. In sentence transformers, this is called multiple negative ranking loss. And it is the same idea that CLIP used to put images and text into one space. Now, there is one catch with this. Say, two customers ask about the same reset procedure and both pairs land in the

03:42 same batch. The loss is going to treat the second copy of that passage as a wrong answer for the first question. So, you’re teaching the model that the right answer is wrong in this case. The fix is a batch sampler that is never going to put the same text twice in one batch, which Sentence Transformers calls the no duplicate sampler. Now, the second thing to get right is the prompt. Embedding Gemma 2 expects a short task

04:12 prompt in front of a search query and a title and text format for documents. So, whatever format you train with is the format your search code needs to use it. Think about this as a prompt template that people use for large language models. The prompt template that you use for training should be exactly the same as the one that you’re using in production or inference. Okay. So, now let’s talk about what actually gets trained because this is the most interesting part of the video.

04:44 Instead of updating all the weights, we are going to use a LoRA, which adds small trainable matrices next to the existing layers and keeps the original weights frozen. LoRA adapters are extremely fast to train, so you don’t really need to retrain the entire network. In my runs, that was less than 2% of the weights, so you can save a small adapter file instead of a full copy of the model. Now, in the last video we saw that text, images, video, and audio all go through

05:16 the same backbone for Embedding Gemma 2. So, if you train the backbone on audio, you are also changing the part that handles your photos and your text, and we need to check what actually is going to be the cost of that. The last concept is how you measure all of this. So, training loss going down only tells you that the model got better at the examples it was trained on, but what you want to care about is the search quality on questions that it

05:47 hasn’t seen. Otherwise, it’s going to be just overfitting. So, you keep a test set aside and you split it by source. If the same article has passages in both train and test, the model has effectively seen the answer already. So, you want to split by articles, by videos, or by recording. Now, this also raises a question for experiments, which is whether the model learns a general skill or only learns your labels. Okay,

06:19 so that is enough theory. Uh now, let me show you the notebook, which is built on Unsloth embedding Gemma 2 notebooks, and I ran everything on an A100 uh GPU in Colab. The link is going to be in the video description. The first example fixes the weak spot from the last video. Um if you haven’t seen it, I highly recommend to watch that video. The example was everyday sounds. So, I used a dataset called ESC-50, which has short clips of 50 everyday

06:50 sounds. Things like dog barking, rain, or a chainsaw. Now, each sound uh name becomes a sentence. So, for example, the sound of a chainsaw. So, classification becomes search in this case because the model embeds the clip, embeds the label sentence, and picks the closest one after training. So, before training, the model picked the right sound first only about one time in four. So, you’re looking at about 25% accuracy. I also noticed that

07:22 the model’s built-in classification prompt did even worse than writing the label as a search query. So, a better prompt is not going to fix this, which is where the training actually comes in. All right, [snorts] so how exactly is this trained? The training pairs are label sentence and the audio clip. Now, this is exactly where the duplicate trap is important to remember because every label has many training clips. So, without the no duplicate sampler, the

07:52 same label would land in a batch. I added LoRA to the audio encoder and the backbone. And the training took about 7 and 1/2 minutes on the A100 GPU. So, the thing to watch is the top result for clips the model had never heard during training. It went from about 1 in 4 to about 2 in 3. And the most interesting thing was that there were some sounds that were never corrected to be correctly identified

08:23 almost every time. For example, brushing teeth sound, right? So, you can see that training a LoRA adapter on top of the model weights actually gives you a huge boost in accuracy. Okay. Now, let’s talk about whether it learned to hear or only learned the labels. So, I ran it again, but this time I hid 10 sounds completely from training and tested on those 10. On the sound it trained on, it improved just like before. But, on the 10 sounds

08:54 it never saw, it didn’t get better. So, the model learns the labels you give it, not hearing in general. Which means if you fine-tune your own application, every label you care about needs to be in the training data. It cannot just generalize. Now, let’s talk about what this cost the rest of the model. That is, what is the actual impact if you try to fine-tune it. So, I loaded the full model with and without the adapter and ran three checks, which are photo search, voice

09:26 search, and general text search. Photo search and voice search did not move at all, but general text search dropped by about four points, which is the the of changing the shared backbone. So, if your application also relies on general text search, I would test it after every fine-tune. Now, the first time I ran this check, voice search dropped to almost zero, which looked like the model had forgotten speech. So, I went through it step by step and the adapter weights were loaded exactly

09:58 right. The problem was the processor that is saved with the adapter. Now, in my case, it was cutting every audio clip down to a few milliseconds, so every sound looked kind of the same to the model. The fix was just one line that reuses the processor from the base model. So, if you save and reload an audio adapter, at least with the version I used, test it again after you reload it. Okay, now let’s talk about the text examples from the demo. So, I took

10:28 transcripts from 120 of my videos and cut them into short passages. Now, I did not have the questions for these passages, so I used Gemini 3.8 flash write one viewer-style question for each one of those passages. That gave me pairs of question and passages that answers it, which is exactly the format we talked about. Then, I split it by video, so the test questions comes from videos that the

10:59 model was not trained on or it hasn’t seen them during the training. Now, before training, the right passages come back first about two times in three. After less than 3 minutes of training, it went up to three in four. General text search dropped by about four points again, which is the same cost as in the sound example that we saw before. So, this is the search from start of the video. And to use it on your own data, you only

11:30 need a file of questions and passages in the same format. So, the answer is yes. A few minutes of fine-tuning made it clearly better on my own sounds and my own videos, but it learned my labels rather than general skills and it gave up a few points on general text search. So, if you’re going to be fine-tuning Gemma 2, I would focus on three things. First one is turning your data into pairs and keeping duplicates out of batches. The second one is testing on

12:01 the data it has not seen. So, you need to split it by source. And the last one is checking what else changed after training because the backbone is shared by every modality. And I think this is one of the most practical things you can do with an embedding model, at least in my experiments. If you have fine-tuned embedding models on your own applications, please do share what worked for you in the comment section below. Links are going to be in the video description. Anyways, I hope you found this video useful. Thanks for watching

12:33 and as always, see you in the next one.