Why You Can't Trust Vector Database Benchmarks

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Why You Can’t Trust Vector Database Benchmarks · Channel: Better Stack · 02:04 · Watch on YouTube

Topics: AI evaluation and benchmarks, Data, databases and embeddings, RAG, search and information retrieval

Summary

Vector database benchmarks can be difficult to trust because measuring search quality requires knowing the exact nearest neighbors, which is computationally expensive at large scale. Qdrant generated a 10-billion-vector dataset with exact results, and the speaker tested a slice of it across Qdrant, Milvus, Elasticsearch, and PGVector. In that test, Elasticsearch did not exceed 95% accuracy, while Qdrant and Milvus reached 98%; Qdrant was described as both quick and the most precise. The broader takeaway is to be skeptical of vendor-published benchmarks and test systems independently.

Key points

Vendors often publish benchmarks in which their own products come out on top.
Benchmarking vector search is difficult, and public benchmarks may rely on shortcuts.
Recall measures how many true nearest neighbors a search engine returns.
Calculating recall requires knowing the exact nearest neighbors in advance.
Qdrant generated a 10-billion-vector dataset using substantial GPU compute.
The speaker tested only a slice of the dataset because the full dataset is 24 terabytes.
The test compared Qdrant, Milvus, Elasticsearch, and PGVector.
Elasticsearch did not get past 95% accuracy in the test; Qdrant and Milvus reached 98%.
The speaker recommends testing vector databases independently rather than treating vendor benchmarks as definitive.

Chapters

Start	Chapter	Summary
00:00	Why vector database benchmarks are hard	The speaker explains why vendor benchmarks can disagree and introduces the challenge of measuring recall accurately.
00:30	Building a large exact-results dataset	The video describes the computational cost of finding true nearest neighbors and Qdrant’s 10-billion-vector dataset.
01:02	Testing four vector search engines	The speaker compares Qdrant, Milvus, Elasticsearch, and PGVector on a slice of the dataset and discusses their accuracy.
01:33	The case for independent testing	The speaker emphasizes the value of a large benchmark dataset and urges viewers to be skeptical of vendor-published results.

Quotes

benchmarking vector search, honestly, is genuinely difficult. — 00:00

you have to know what the right answer is — 00:30

Qdrant and Milvus both reach 98%. — 01:02

we should all test them for ourselves to get to the actual ground truth. — 01:33

Concepts and entities

vector database, recall, benchmark, Qdrant, Milvus, Elasticsearch, pgvector

Full transcript (original language)

00:00 If you’re looking for a reliable vector database for your project, choosing one is so hard. And that is because every vendor has a benchmark page, and every vendor shows they’re winning their own benchmarks. That’s not necessarily because they’re all lying, but it’s because benchmarking vector search, honestly, is genuinely difficult. And almost every public benchmark takes some shortcuts and produces half-baked results. Let’s look at recall, for example. Recall measures the following: Of the true nearest neighbors to a certain query, how many does the engine actually give me back? So, to measure

00:30 recall, you have to know what the right answer is, which means somebody has to have computed the true nearest neighbors first, not approximately, but exactly. But at 10 billion vectors with 120,000 queries, this is over a quadrillion distance computations, 10 to the 15th. And that is precisely why nobody had done it before. But Qdrant went and did the expensive thing. They burnt through a lot of GPU compute to produce this massive data set of 10 billion vectors. So, I tested this tool chain myself, albeit on just a slice of the whole data

01:02 set, because 10 billion vectors is 24 terabytes, and my laptop just doesn’t have that much space. So, I ran it against four engines, Qdrant, Milvus, Elasticsearch, and PGVector. But look at where each engine stops. So, Elasticsearch can’t get past 95% accuracy, no matter how hard you push it. But Qdrant and Milvus both reach 98%. So, they’re good at different things. Elasticsearch is the fastest way to be roughly right, but Qdrant is the only one of the bunch that is quick and still gives you the most precise results. So, the big takeaway here is not the numbers that I showed you in the

01:33 demo, but rather the concept itself that we now have a large enough data set that can be used to accurately benchmark vector databases against each other. And this is also a good reminder of why we should be skeptical of vendor-published benchmarks, because as we just saw in this video, benchmarking vector databases is genuinely a super difficult task. So, I’m not blaming vendors themselves, I’m just stating the reality that nothing is black and white, and we should all test them for ourselves to get to the actual ground truth. So, those are my thoughts on this topic, but what are yours? I’d love to hear your

02:04 insights, so please share them in the comment section down below.