Lakebase Search: Build AI Retrieval for Agents with Lakebase Postgres

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Lakebase Search: Build AI Retrieval for Agents with Lakebase Postgres · Channel: Databricks · 12:34 · Watch on YouTube

Topics: AI agents, Data, databases and embeddings, RAG, search and information retrieval

Summary

The video explains how Lakebase Search brings vector and full-text retrieval into Postgres for AI agents, combining semantic similarity and exact-term matching through hybrid search and reciprocal rank fusion. It describes how Lakebase Vector and Lakebase Text support scalable search alongside operational data, with an architecture designed for cached access, bursty workloads, and compute that can scale to zero. The walkthrough also shows how Spark pipelines can process document changes, generate chunks and embeddings, and synchronize updated content for retrieval. The result is a workflow where agents can search current data without a separate vector database or external synchronization infrastructure.

Key points

AI agents both retrieve and write data, so newly written chats, memories, and outputs need to become searchable quickly.
Lakebase Search introduces Lakebase Vector, compatible with PG Vector types and operators, and Lakebase Text, which adds BM25 scoring and optimized top-K retrieval.
Hybrid search combines vector search for semantic similarity with full-text search for exact terms.
Reciprocal rank fusion combines vector and text result rankings using positions rather than raw scores.
Agents can choose between text, vector, and hybrid search tools, an approach described as agentic search.
Rabbit Cube compresses vectors in the index to reduce their size and speed retrieval while maintaining high search accuracy.
Lakebase separates compute from durable storage, using memory or local disk cache for frequently accessed index pages and allowing compute to scale with demand or to zero.
Document-heavy workloads can use Spark for parsing, chunking, and embedding generation while transactional and search queries continue on separate compute.
A Unity Catalog Delta table with change data feed can trigger a Spark declarative pipeline that processes document changes and synchronizes embedded chunks for search.
The described workflow supports hybrid retrieval of updated documents without a separate vector database or external synchronization infrastructure.

Chapters

Start	Chapter	Summary
00:00	Why agent retrieval needs database-native search	The video introduces agents’ concurrent read and write needs, then outlines the scaling limitations of vector and full-text search in Postgres.
01:33	Lakebase Search and hybrid retrieval	Lakebase Text and Lakebase Vector extend Postgres search, and the video explains hybrid search, reciprocal rank fusion, and agent-selected search tools.
05:12	Scaling vector search and processing documents	The video covers Rabbit Cube and Lakebase’s storage and compute architecture, then introduces Spark-based processing for document-heavy workloads.
08:22	Keeping the search index up to date	A Unity Catalog Delta table, change data feed, and Spark declarative pipeline are used to process document changes and synchronize chunks and embeddings.
10:28	Demonstration and recap	The demo follows a document update through the pipeline into a synced table, where an agent can retrieve the latest content using hybrid search.

Quotes

In practice, we often need both. — 02:36

This is what we call hybrid search. — 03:07

This is what we call agentic search. — 04:40

the entire search index doesn’t need to fit in RAM. — 06:15

No separate vector database and no external synchronization infrastructure to manage. — 09:57

Concepts and entities

Lakebase Search, Lakebase Vector, Lakebase Text, PG Vector, BM25, Reciprocal Rank Fusion, Rabbit Cube, Databricks, Unity Catalog, Spark

Full transcript (original language)

00:00 In this video, I’ll show you how LakeBase Search supports the retrieval needs of AI agents. Then, we’ll walk through an end-to-end AI retrieval pipeline. We’ll use hybrid search and automated embeddings to make everything work together. Agents search a lot, but they don’t just retrieve information, they constantly write new data, too. Chat history, memories, outputs, all of it needs to be searchable the moment it’s written. And agents can run tens or even hundreds of

00:31 queries concurrently. This makes search a critical OLTP workload, one that can’t be siloed away from your database. But, supporting search at this scale comes with challenges. For vector search, PG Vector is a popular choice in Postgres. However, as indexes grow, keeping them in memory becomes difficult. And when searches require frequent disk access, performance can degrade significantly. One option is to add a dedicated vector database.

01:02 But then, you’re managing two separate systems, and you need a data synchronization pipeline to keep them consistent. For full-text search, Postgres already provides built-in support through TSVector. However, its native ranking functions don’t implement BM25. BM25 gives us more sophisticated relevance scoring. It considers how distinctive a term is across documents, how frequently it appears, and the length of each document.

01:33 This is where LakeBase Search comes in. LakeBase Search introduces two new Postgres extensions, LakeBase Text and LakeBase Vector. Let’s start with LakeBase Vector. It builds on PG Vector and takes advantage of the LakeBase architecture. And, importantly, it remains fully compatible with PG Vector’s types and operators. Similarly, Lake Base Text extends Postgres full text search. It adds native BM25 scoring and optimized top K

02:05 retrieval. This helps improve relevance and performance as data sets grow. But, what’s particularly interesting is the underlying architecture. With Lake Base Search, search lives directly alongside your operational data. You can use a single SQL query to search, filter, and join with operational data. And all of these benefits from Lake Base’s decoupled storage and compute architecture, allowing search workloads to scale.

02:36 Now, scaling search is important, but we also want retrieval to be accurate. Vector search is great at capturing semantic similarity. It understands the meaning and intent behind a query. Full text search, on the other hand, is great for exact matches. Things like product names, references, and specific keywords. In practice, we often need both. We want semantic search to understand what the user means, and full text search to capture exact terms.

03:07 With Lake Base Search, we can combine both approaches directly in SQL. This is what we call hybrid search. Now, how do we combine the results? One common approach is reciprocal rank fusion, or RRF. RRF merges the two rank lists. But, instead of using their raw scores, it uses their positions. This avoids problems caused by different scoring scales. So, if a result ranks highly in both vector and text search, it gets boosted

03:38 toward the top, even if it wasn’t the number one result in either list. Let’s look at a few examples, all powered by Lake Base Search. First, let’s start with a simple search bar for tech products. When users type into a search bar, we often want to provide suggestions immediately. For this, Full Text Search is a great fit. It’s fast and it can match product names and references. Now, when the user actually runs the

04:08 search, we want to go further. We don’t just want exact keyword matches, we also want to capture the meaning behind the query. And that’s where hybrid search comes in. As you can see, the results are influenced by both semantic similarity and keyword matches. Now, this is already useful, but with agents, we can take it one step further. Instead of having the application decide which search method to use, we give the agent access to the search tools directly. Text Search, Vector Search,

04:40 and Hybrid Search. The agent can then decide which tool to use and when to use it. This is what we call agentic search. Let’s look at an example with a support agent. I’ll ask it to find a sound bar that’s compatible with a TV I already own. First, the agent searches its memory. It needs to find which TV I purchased in the past. Then, it runs a combination of vector, text, and hybrid searches, using those results to build its final

05:12 response. And, as you can imagine, performance matters a lot here. The agent might need to perform several searches before answering. So, both retrieval accuracy and query latency directly affect the user experience. Now, let’s look at how LakeBase supports these workloads at scale. To deliver fast and accurate vector search, LakeBase Vector uses an algorithm called Rabbit Cube. Rabbit Cube compresses vectors inside

05:44 the index. This reduces their size and enables faster retrieval while maintaining high search accuracy. Another important aspect is the Lake base architecture. Lake base separates compute from durable storage. Frequently accessed index pages are served from memory or local disk cache. When the page isn’t available in the cache, the request goes to the page server. Which can access durable storage when needed. The key benefit is that the

06:15 entire search index doesn’t need to fit in RAM. And this architecture also helps with bursty workloads. When traffic increases, Lake base can automatically scale compute to meet demand. And when the workload becomes idle, compute can scale to zero after a configured timeout. The index remains in durable storage ready for the next query, so you can eliminate idle compute costs without having to rebuild the index when compute starts again.

06:46 So far, we’ve seen how Lake base search serves retrieval workloads for agents. But remember, agents don’t just read data. They also write it. Chats, memories, states, and outputs. And all of that information needs to become searchable as soon as it’s written. For some types of data, this is straightforward. When you insert a new row in Postgres, you can query it immediately. But things become more complex with document heavy workloads. For example, building a

07:18 knowledge base or keeping an existing knowledge base up to date as documents change. These workflows require several steps. Parsing documents, splitting them into chunks, generating embeddings, and indexing the results. These tasks can be expensive. And ideally, we don’t want them competing with transactional queries for resources. This is where ELT app comes in. Lakehouse share the same underlying storage layer.

07:50 This allows us to use different compute engines for different workloads. We can offload heavy processing to Spark. For example, document parsing and embedding generation using AI functions. Meanwhile, Lakehouse continues serving transactional and search queries without having to run those expensive transformations on its own compute. And the same architecture works for different scenarios. Whether we’re indexing an entire document collection for the first time or continuously

08:22 updating the search index as agents modify documents. Let me show you how this works with an example. Imagine we have an agent that can create and update documents. Over time, those documents will change, but the agent still needs to retrieve the latest information. So, the question is, how do we keep the search index up-to-date automatically? With native Databricks features, we can build this workflow without external infrastructure. Here’s how it works. First, our

08:53 documents are stored in a Unity Catalog Delta table. We enable Lakehouse change data feed or CDF. This allows us to track every insert, update, and delete. Next, we use a Lakehouse job with a table update trigger. Whenever the source table changes, the job triggers a Spark declarative pipeline. Inside that pipeline, the first step is to create a streaming table. This streaming table applies the changes and and reconstructs the latest version of our documents in

09:25 Delta. Then, we create a materialized view, and this is where we perform document processing. We use AI prep search to semantically chunk the documents and AI query to generate their embeddings. Finally, we create a sync table. This brings the embedded document chunks into Lakehouse as a read only possible stable. Once the data is synchronized, Lakehouse search can use it for hybrid search. And the best part is that the workflow stays synchronized automatically

09:57 from Lakehouse to the Lakehouse through the Spark pipeline and back to Lakehouse through sync table. So, when an agent updates a document, the pipeline processes the change. And the updated content becomes searchable once synchronization completes. No separate vector database and no external synchronization infrastructure to manage. Let’s see this in practice. Here, the agent can create and modify documents. I’ll ask it to update a guide for purchasing head foods. Specifically,

10:28 I’ll ask it to add a new section about water resistance. After a few seconds, the agent has updated the guide. Behind the scenes, this means the document was updated in the database. Now, let’s see what happens next. First, the document update should trigger our Spark declarative pipeline. Let’s navigate to jobs and pipelines. Here, we can see that the pipeline has been triggered and it’s now running. Let’s open it and take a closer look.

11:00 Our source data comes from a Delta table. A streaming table processes the document changes and the output is a materialized view containing the document chunks and their embeddings. Now, let’s look at the code. Here, we can see how the pipeline is defined and how it uses two native SQL functions, AI prep search for semantic chunking and AI query for generating embeddings. Next, let’s navigate to Unity Catalog. Here, we can inspect the resulting materialized view

11:31 and see the document chunks and generated embeddings. From here, we can create a sync table. In my case, I’ve already created it and the data has been synchronized. So, let’s head over to the Lakehouse interface. First, I’ll select the right project, then the right schema, and finally, I’ll open the synced table. Here, we can see the chunked documents together with the embeddings. And because this data is now in Lakehouse Postgres, we can use Lakehouse search

12:02 directly. Both vector search and full-text search are available, and we can combine them for hybrid retrieval. So, the agent can retrieve the right documents, including the latest updates. So, to sum up, Lakehouse Postgres provides a unified back-end for mission-critical OLTP workloads, including search and retrieval for AI agents. And because agents need both transactional and analytical capabilities, Lakehouse lets us use the right engine for each task. Lakehouse

12:34 for serving queries and the Lakehouse for heavy data processing and AI transformations, all without stitching together multiple platforms or actively managing separate CDC and ETL pipelines. You can find the notebook example in the description. Thank you.