This 150M Model Claims #1. I Tested It.

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This 150M Model Claims #1. I Tested It. · Channel: Prompt Engineer 48 · 07:56 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, Local and open-source models, RAG, search and information retrieval

Summary

The video examines the model card for Ultraert Embed 150M, which reports an average MTEB score of 67.24 and claims a leading result among dense models under 200 million parameters. The creator reviews the reported benchmarks and notes that the results are unverified and author-reported. In a small local search test using 11 questions and 21 chunks, Ultraert ranked the correct chunk first less often than the two comparison models and was slower per query. The creator emphasizes that the test did not evaluate the long-document or code-retrieval tasks where the model’s reported advantages are strongest, and recommends testing it on your own data.

Key points

Ultraert Embed 150M is an embedding model, not a chat model; it maps text to vectors for tasks such as search, classification, and clustering.
The model card reports 768-dimensional embeddings, a 32K-token context, and an MTEB English v2 average of 67.24 across 41 tasks.
The reported MTEB lead over the next-listed model is less than one point.
The model card’s benchmark figures are author-reported, and the leaderboard entries discussed are not verified.
The creator ran the model locally on an RTX 4060 laptop GPU with 8 GB of VRAM; peak VRAM use was 0.68 GB.
The local test used 21 chunks from the model card and 11 questions; Ultraert got the correct chunk first for five questions, compared with six for each of the other two models.
In that test, Ultraert was slower per query than GTE Modern BERT Base and BGE Base v1.5.
The test did not stress 32K documents or code retrieval, which are areas where the model card reports strengths.
The creator suggests testing Ultraert on long documents or code, while short-document users may find the comparison models suitable.

Chapters

Start	Chapter	Summary
00:01	The model and its claims	The creator introduces Ultraert Embed 150M and checks the model card’s reported size, context length, license, and MTEB score.
01:36	Reviewing the benchmark tables	The video compares reported results across tasks and notes that the benchmark numbers are author-reported and unverified.
03:45	Training claims and local setup	The creator summarizes the authors’ training claims and explains the local GPU setup and retrieval test design.
05:20	Test results and who should try it	The small test favors the comparison models, but the creator notes its limitations and recommends testing Ultraert on long documents or code.

Quotes

the result was not what the card suggests. — 00:01

So almost nobody has independently tested it yet. — 01:36

they should be checked on your own data. — 03:13

So on this test Ultra did not win. — 05:54

the thing ultra bird is built for long context never got used. — 06:24

Concepts and entities

UltraERT Embed 150M, MTEB, GTE ModernBERT Base, BGE Base, Sentence Transformers, DMEIL Lab, RunPod

Full transcript (original language)

00:01 This is the model card of a 150 million parameter embedding model and it says 67.24 on MTEB, the highest of any dense model under 200 million parameters. All right, in this video we will check that claim on the real page and then I ran my own search test on this 8GB laptop GPU and mind you the result was not what the card suggests. Let’s go ahead and see. So this is the blog post on hugging face titled Ultraert one encoder for classification and retrieval. It’s a

00:32 community article from DMEIL lab published October 14th, 2026. And you can see it has 37 upvotes on the right. In the TLDDR they say it is a family of English encoders 150M and 380M with a native 32K context plus ultra embed 150M, a readytouse embedding model built on top. Everything is Apache 2.0. Now this is an encoder not a chat model. It does not write text. It turns text into a vector so that you can search, classify and cluster. That is why it’s

01:04 small and that is why it can run on a laptop like this one. Now let’s open the model card for Ultraert Embed 150M. On the right you can see the model size 0.2B parms, the safe tensors file and the license Apache 2.0 up on top. If you look at the table in the middle, the embedding model has 768 hidden size and a 32k context. The text says it maps text to a 768 dimensional vector and takes inputs up to 32,000 tokens. Here’s the thing to

01:36 notice though. It’s very new. You can see 125 downloads and 27 likes on this page. So almost nobody has independently tested it yet. And the card itself is the source of these numbers. in the tab files and versions. The whole thing is one safe tensors file of about 151 million parameters. So loading it is simple. Hold on to that number 67.24 because I’m going to come back to it. Let’s read the actual table. This is MTB English version 2 41 tasks. The first row Ultra Bird embed 150M average 67.24.

02:10 Next one is Tarka embedding 150 MV1 at 66.39. And then GT modern bird base at 66.26. So the lead is under one point over the second and 1.0 over the third. It’s ahead but not by a mile. Now look at the columns in classification. Ultra scores 81.74 and the distilled F2 LLM. V2 160M scores 87.13 which is the bold number in that

02:41 column. Tarka gets 86.28. So on classification it’s actually behind where it wins is retrieval at 5720 and summarization at 34.89. Now go down to the code table. MTEBB code version 1 average 73430 and KOR is 72.73. On the apps column it’s 685D while GTE modern bird base is 57.72. So that’s a big gap on code search. And

03:13 this one is the one I care about. Long embed 7930 average and Pascal retrieval 99.00 with documents up to 32,768 tokens. The next one, Beco only goes to 8192 and gets 73.19. One more honest thing, the card says these are the author’s own reported numbers. The verified flag on the leaderboard entries is false. So they should be checked on your own data. That is exactly what I did.

03:45 The paper is on RXE 2610 1528 ultra beyond the limit of bird pre-training. And in the blog they make two claims. You can see here encoders stop improving after 1 to2 trillion tokens even while the loss keeps going down. And the masking ratio decides what the encoder is good at. Low masking wins classification, high masking wins retrieval. And the blog says they mix five and 50% masking in every batch. And the card says the embedding model is trained with contrastive losses on public pairs only without distillation

04:16 from a larger embedding model. Now let’s run it. Mind you that everything here is on my local system and RTX 4060 laptop GPU with 8 GB and almost 7 GB of that was already used by other things. The card says you need Sentence Transformers 6 or newer, Transformers 5 or newer and trust code equals true. So I set up a fresh virtual environment and one catch pip gave me a CPU only torch. So I swapped it to the CUDA build. After that it loads in FP16

04:49 and you can see peak VRAM was 0.68 GB. So the model itself is tiny. The code is simple. For queries you pass the instruction prefix instruct. Given a question retrieve passages that answer it query and the documents go in raw with no prefix. Here is my test. I took the model card itself, cut it into 21 chunks, and wrote 11 questions where I know which chunk has the answer. Things like, what is the

05:20 maximum input length in tokens or how do I enable flash attention? I ran three models, Ultra Bird, Embed, 150M, GTE, Modern Bird Base, and BGE Base and V1.5. And mind you, it is a small test, 11 questions. Here are the results. Ultra Bird got the right chunk first in five of 11. So recall at one is 0.455. Recall at three is 0.727. GTE modern bird base got six of 11 first. 0.545 and recall at 3 0.818. BG

05:54 base got the same.545 and 818 and the best MR 0.727 against 0.647 for Ultraert. So on this test Ultra did not win. It was one question behind at rank one and it was also the slowest per query. I ran the whole thing twice. 55 and 74 milliseconds median for Ultra Bird, 36 and 44 GTE, 22 and 25 for BGE. So what does that mean? It doesn’t mean

06:24 the card is wrong. My corpus is 21 chunks, 17 of them under 200 tokens. And the four big table chunks were cut at the 512 token limit I set. So, the thing ultra bird is built for long context never got used. I also never tested code. The card’s claim is strongest on exactly the two things I did not stress 32k documents and code retrieval. And I did not profile the speed gap. So, I can’t tell you why it is slower here. It needs trust from my code which is custom model code. So, if you search short

06:55 English paragraphs ABG base or gte modern base is fine and faster in my test. If your documents are long or you search code, this is the one I would test first on your own data. And since the numbers are the author’s own, test it on your own data before you trust the 67.24. So who is it for? You have a laptop or a CPU box. Your documents are long and you want one model that handles retrieval, classification, and similarity with an instruction on the query. That is what

07:25 this card describes. If you have short documents, you can stay with what you have. The next step I would do is a repo of my own code with whole files embedded at 8K and 16K tokens and compare the three again. If you want that, let me know in the comments. If you’re looking for cheap and reliable GPUs, I use RunPod and the link is in the description. So, in this video, we looked at the real model card, the real tables, and I ran my own small search test. Try it on your own documents and let me know how it goes. Model used DMS

07:56 Lab Ultraird Embed 150M. I’ll see you in another one.