Qdrant Just Gave Away 10 Billion Vectors For Free (FineWeb 10b)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Qdrant Just Gave Away 10 Billion Vectors For Free (FineWeb 10b) · Channel: Better Stack · 12:33 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, RAG, search and information retrieval

Summary

The video introduces Qdrant’s FineWeb 10B dataset: 10 billion real-world documents embedded as dense and sparse vectors, with exact ground truth for 120,000 queries. Its scale and brute-force ground truth aim to address weaknesses in common vector database benchmarks, while the open-source Supernova toolkit lets users reproduce tests or benchmark their own data. The presenter tests four databases on a one-million-vector slice and finds different trade-offs in speed and recall, while stressing that the results do not establish how they perform at 10 billion vectors. The main takeaway is to treat vendor benchmarks cautiously and test systems on representative data.

Key points

The dataset contains 10 billion FineWeb documents, totaling 24.5 terabytes of embeddings, and is available free on HuggingFace.
Qdrant released the Supernova benchmark toolkit under the Apache 2.0 license for embedding generation, database loading, and load testing.
Small benchmarks can miss memory pressure, index build times, and tail latency that emerge at production scale.
The dataset includes dense and sparse embeddings generated with GTE multilingual base, enabling hybrid retrieval benchmarks.
It includes 120,000 queries: 100,000 dense, 10,000 sparse, and 10,000 filtered queries, with the 10,000 nearest documents recorded for each.
The ground truth was computed through brute-force comparisons, requiring over a quadrillion distance calculations.
The presenter tested Qdrant, Milvus, Elasticsearch, and pgvector on a one-million-vector slice, computing separate exact ground truth for that subset.
In the slice test, Elasticsearch was fastest at roughly 94% accuracy, while Qdrant and Milvus reached 98%; these results may not hold at 10 billion vectors.
The presenter cautions that the dataset and benchmark choices still reflect vendor decisions, and recommends testing databases on one’s own data.

Chapters

Start	Chapter	Summary
00:00	Why vector database benchmarks are difficult	The video introduces the problem of vendor benchmarks and explains why scale and exact ground truth make vector search difficult to benchmark.
03:23	What FineWeb 10B contains	The presenter describes the corpus, its dense and sparse embeddings, its query sets, and the exact nearest-neighbor ground truth.
06:00	Caveats and the Supernova toolkit	The video notes that this is still a vendor-created benchmark, explains why its ground truth is verifiable, and introduces the open-source toolkit.
07:37	Testing a slice of the dataset	The presenter explains why published 10-billion-vector ground truth cannot be used for a subset and outlines a test that computes exact ground truth for the slice.
09:19	Results and takeaways	The presenter compares four databases across recall and latency settings, cautions against extrapolating the slice results, and recommends testing benchmarks independently.
12:03	Closing	The presenter invites viewers to share their thoughts and closes with a request to like and subscribe.

Quotes

24 and a half terabytes of embeddings and it’s completely free to download from HuggingFace — 00:34

somebody has to have computed the true nearest neighbors first, not approximately, but exactly. — 02:20

This is still a vendor benchmark. — 06:00

So I ran each engine on five settings instead of one. And that gives you a curve. — 09:52

we should all test them for ourselves to get to the actual ground truth. — 11:32

Concepts and entities

Qdrant, FineWeb 10B, Supernova, GTE multilingual base, FineWeb, MS MARCO, Milvus, Elasticsearch, pgvector, Hugging Face

Full transcript (original language)

00:00 If you’re looking for a reliable vector database for your project, choosing one is so hard. And that is because every vendor has a benchmark page and every vendor shows they’re winning their own benchmarks. That’s not necessarily because they’re all lying, but it’s because benchmarking vector search honestly is genuinely difficult and almost every public benchmark takes some shortcuts and produces halfbaked results. But Quadrant just published the ultimate solution to this problem and it’s a benchmark data set with 10

00:34 billion vectors in it. 24 and a half terabytes of embeddings and it’s completely free to download from HuggingFace and alongside it they also open sourced the entire tool chain they used to build it under the Apache 2.0 license. So, in today’s video, we’re going to take a look at this benchmark data set, see how it works, and we’re going to use it to benchmark some of the most popular vector databases to find out what the real benchmarks actually tell us. It’s going to be a lot of fun. So, let’s dive into it.

01:09 So, there are two things that are genuinely difficult when benchmarking vector databases. The first one is scale. Quadrant’s own previous benchmark numbers ran on a data set called DBPeddia OpenAI 1 million which is 1 million vectors large and deep image 96 which is 10 million. And if you’re running a 100 million vectors in production, a 1 million vector benchmark tells you essentially nothing because at 1 million everything fits in RAM and everything is fast when it fits in RAM. The behavior that actually stress tests

01:44 your system at scale is the memory pressure and the index build times and the tail latency when the index starts paging off disk. But none of that really shows up with a small sample size. So you are measuring a completely different setup from the one you’re going to be deploying on production. And the second problem is a bigger one and it’s ground truth. Let’s look at recall for example. Recall measures the following. Of the true nearest neighbors to a certain query, how many does the engine actually give me back? So to measure recall, you

02:20 have to know what the right answer is. Which means somebody has to have computed the true nearest neighbors first, not approximately, but exactly. And that means every single query compared against every single vector in the corpus with no index, no shortcuts, and no approximation. And at 1 million vectors, it’s quite doable. You can basically do it on a laptop. But at 10 billion vectors with 120,000 queries, this is over a quadrillion distance

02:52 computations. 10 to the 15. And that is precisely why nobody had done it before. So most benchmarks do one of two things instead. Either they compute approximate ground truth which means you’re now measuring one approximation against another approximation and calling the result accurate or they use synthetic vectors randomly generated which don’t have the clustering structure the real embeddings have. So the search problem itself is artificially easy. But

03:23 Quadrant went and did the expensive thing. They burned through a lot of GPU compute to produce this massive data set of 10 billion vectors. So what’s actually in it? Well, the corpus is fine web which is hugging faces common crawl data set and it contains real web pages and real text. They took 10 billion documents out of it and embedded every single one twice using a model called GTE multilingual base once in a dense vector which encodes meaning. So that

03:55 for example car and automobile end up near each other and one into a sparse vector which encodes actual terms closer to how a traditional keyword index works. And having both from the same model over the same documents means you can benchmark hybrid retrieval without changing anything else about the data. And then there’s the ground truth. 120,000 queries total. 100,000 of them are dense queries taken from MS Marco which is a public collection of real

04:27 search queries typed by real people. Then 10,000 sparse queries and 10,000 filtered queries. And ground truth is really important because filtered vector search meaning find me similar documents but only once tagged with X is what a lot of engines fail at. But it is also the thing that basically every real application does and almost nobody benchmarks it because generating filtered ground truth is even more work than generating the normal kind. So for

04:57 every query they recorded the 10,000 closest documents in order. So why does that matter? Well, because different applications ask for different amounts. A chatbot doing retrieval might pull the top five documents to stuff into a prompt. A search results page might show 20 results. A serious search system often pulls a thousand candidates and then runs a slower, smarter re-ranking model on top of them. Those are genuinely different questions and

05:28 normally each one needs its own groundtruth run. But if you already have the top thousand in order, you get all of those for free. If you want to know the true top 10, it’s the first 10 items on the list. The true top 100, well, those are the first 100. You just cut the list off wherever you need it. And because it’s already sorted best first, what’s left is exactly the answer you’re looking for. And that’s why it took an insane amount of compute to generate this data set. Computing the ground truth took over a quadrillion distance

06:00 calculations doing it once at depth 1,000. So this might be the most precise ground truth data set out there. But let me be clear here because I don’t want to overhype it as well. This is still a vendor benchmark. Quadrant themselves is a vector database company and Quadrant chose this corpus and Quadrant also chose the embedding model. Quadrant chose also the query set and built the harness. All of those choices shape the results and different embedding model

06:31 might give you a different difficulty curve. But here’s what makes it structurally different from a usual data set. The ground truth is either correct or it isn’t. It’s a brute force computation with no free parameters. There’s no turning knob, no clever configuration, so there’s nowhere to hide. Anybody with a GPU budget can take the data set, recomputee the exact nearest neighbors themselves, and check whether Quadrron’s answers match. And that’s fundamentally a stronger position than asking to trust a number on a

07:03 marketing page. And the tooling is open source. The framework is called Supernova. It’s Apache 2.0 O on GitHub and it does embedding generation, database loading and load testing. So you aren’t limited to reproducing their benchmark. You can point it at your own corpus and build the same kind of benchmark on your own data which is the thing you actually want because it’s your data and not fine web. So, I tested this tool chain myself, albeit on just a slice of the whole data set because 10 billion vectors is 24 terabytes and my

07:37 laptop just doesn’t have that much space. And this is something I should disclose as well because since I’m not running the whole benchmark and if I’m testing Quadrant, for example, my ground truth numbers against Quadrants will be very inaccurate. And that’s because Quadrron’s published ground truth was computed across all 10 billion documents. The moment you take a slice, the ground truth doesn’t apply to you anymore because the true nearest neighbors for most queries live in the billions of documents you didn’t download. So if I score my slice against quadrants published answers, it will

08:11 look catastrophically broken and none of it means anything. So you have to take in mind that on a subset you compute your own ground truth over exactly the vectors you loaded and that’s what Nova BF is for. You can run it and still get an exact brute force answer and the recall measured against it is completely valid. It’s just answering a different question. You won’t get an answer for did this engine match quadrant’s 10 billion vector result, but instead you’ll be measuring did this engine’s fast index find what an exhaustive

08:45 search over my million documents finds. And that tells you whether the index works. So basically what I’m trying to say is that computing brute force is a hard computationally expensive task and not all consumer devices can do it. But for testing purposes, we can still compute it on just a slice of the data set as a test. So anyway, I made a oneshot script that you can run and it will automatically spin up a docker instance of quadrant, milvous, elastic and pg vector databases. And then it pulls a slice from hugging face,

09:19 regenerates the query embedded, computes exact ground truth over your slice, and then for each database, in turn, it spins up a container, loads the same million vectors, and sweeps the search effort setting across five values while measuring recall and latency at each one. So, same ground truth for every engine, one at a time, so they’re not fighting over your CPU. So I ran it against four engines. Quadrant, Milvvis, Elastic Search, and PG Vector. And one number per engine would be useless here because every one of these has a setting

09:52 that controls how hard it looks before answering. So if you look less, you’re going to be fast, but you’re going to miss things. And if you look harder, you’re going to be slower, but you’re going to find more. So I ran each engine on five settings instead of one. And that gives you a curve. And that curve lets you ask the real question which is at that same accuracy which one is faster and at 94% accuracy elastic search answers in 7 1/2 milliseconds. Quadrant is 9 1/2 mil 12 1/2 and PG vector is 13. And elastic search winning

10:28 actually surprised me because this is an engine people think of as a keyword search with vectors bolted on. So, it isn’t necessarily a vector database first, but it did beat all three other databases, but look at where each engine stops. So, Elastic Search can’t get past 95% accuracy no matter how hard you push it. But, Quadrant and Milvas both reach 98%. So, they’re good at different things. Elastic Search is the fastest way to be roughly right, but Quadrant is

11:00 the only one of the bunch that is quick and still gives you the most precise results. And I have to say again this is just a million vectors not 10 billion. So whether these curves hold at thousand times the scale is not guaranteed. The results might be totally different if we would run this for 10 billion vectors. So there you have it folks that is Quadrron’s new fine web 10 billion vector data set. So the big takeaway here is not the numbers that I showed you in the demo, but rather the concept itself that we now have a large enough

11:32 data set that can be used to accurately benchmark vector databases against each other. And this is also a good reminder of why we should be skeptical of vendor published benchmarks. Because as we just saw in this video, benchmarking vector databases is genuinely a super difficult task. So I’m not blaming vendors themselves. I’m just stating the reality that nothing is black and white and we should all test them for ourselves to get to the actual ground truth. So,

12:03 those are my thoughts on this topic, but what are yours? I’d love to hear your insights. So, please share them in the comment section down below. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video. And also, don’t forget to subscribe to our channel. This has been Andress from Better Stack and I will see you in the next videos. [music]

12:33 >> [music]