Building Production-ready AI Agents on Databricks | Agent Bricks Demo

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Building Production-ready AI Agents on Databricks | Agent Bricks Demo · Channel: Databricks · 06:49 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Data, databases and embeddings

Summary

The demo walks through a production-ready agent lifecycle on Databricks: index data, build and trace an agent, register it, evaluate it, and deploy it. It shows how MLflow traces expose an individual request’s retrieval, model calls, latency, token count, and cost. The evaluation example separates answer relevance and safety from a custom check that the agent stays on the product catalog, revealing when it answers a question correctly but goes off topic. The same evaluation scores can be used in development and production to monitor agent behavior after deployment.

Key points

A vector search index can use managed embeddings over a Delta table, stay in sync with data changes, and be governed by Unity Catalog.
The agent-as-code pattern makes an agent reviewable, versionable, and deployable.
MLflow tracing captures request details such as retrieval, tool calls, latency, and tokens.
The agent is registered in Unity Catalog as a governed version artifact and evaluated with MLflow before deployment.
The demo lifecycle is: index, build, trace, register, evaluate, and deploy.
Traces help diagnose whether a production issue comes from retrieval or the prompt.
The example agent answers that Paris is the capital of France, even though the question is outside the Databricks product catalog.
Evaluation distinguishes relevance and safety from whether an answer stays on catalog; the example scores 100% on relevance and safety and 80% on the catalog check.
The same evaluation scores can run in development and production to monitor behavior after shipping.

Chapters

Start	Chapter	Summary
00:00	Agent lifecycle and setup	The presenter outlines the steps for indexing data, building and tracing an agent, registering and evaluating it, and deploying it.
01:35	Inspecting agent traces	The demo examines a single request trace, including its retrieval, model call, latency, token count, and cost.
03:08	Using traces to diagnose issues	The presenter shows how traces reveal agent behavior and help identify whether retrieval or prompting caused a production problem.
04:09	Evaluating agent behavior	An MLflow evaluation compares relevance and safety with a custom check for staying on the product catalog.
06:18	Monitoring in production	The presenter explains that the same evaluation scores can be used in development and production to track agent behavior.

Quotes

you don’t have to set up a separate embedding service — 00:00

So we index, build, trace, register, evaluate, deploy. — 01:35

instead of squinting at logs and guessing. — 02:37

does the agent stay on our product catalog or does it wander off topic? — 04:42

This is the difference between hoping your agent behaves and then knowing it does. — 06:18

Concepts and entities

Databricks, Agent Bricks, Unity Catalog, Delta table, MLflow, mlflow.genai.evaluate, índice de busca vetorial, agent as code

Full transcript (original language)

00:00 [music] >> So step one, we create a vector search index over a Delta table with managed embeddings. And managed is the keyword here because you don’t have to set up a separate embedding service and the index itself keeps in sync with the table as the data as the data changes. And like everything else, it’s governed by Unity Catalog.

00:33 Step two, we load the agent from a file using the agent as code pattern and we query it. Keeping the agent in code rather than clicking it together is what makes it reviewable, versionable, and deployable later. Step three is MLflow tracing, which captures exactly what happened on every request. So this includes the retrieval, the tool

01:04 calls, the latency, and the tokens. Step four, we register the agent to Unity Catalog as one governed version artifact, which is a single source of truth for what’s deployed. Step five, we evaluate it with MLflow. And step six, we deploy it to a real-time endpoint with a review app so humans can give it feedback.

01:35 So we index, build, trace, register, evaluate, deploy. Rolls off the tongue, right? That’s the whole life cycle and it’s the same shape whether you’re building a weekend prototype or something that serves real traffic. So I’ll show you two moments in the loop that are super important. So, going over here, this is one of our experiments and we are looking at the

02:06 traces of our agent. So, this would be the trace of a single request. On the left, you can see the agent’s execution laid out step-by-step. So, we have the retrieval step and then the call to the language model. Here, we have the latency, the token count, and even the cost per request. And then, this is how you can see what

02:37 your agent did span by span instead of squinting at logs and guessing. So, I chose this example on purpose. So, with the lab example, we’ll be um using a database of the Databricks product catalog. Um but the question over here is what is the capital of France? Which arguably has no nothing to do with Databricks product product catalog.

03:08 But if you see what the agent did, it answered honestly, so saying capital of France is Paris. But then, it steers back to what it’s actually for. And the trace shows you the retrieval that ran and how the model responded. Um and we can see that the whole thing took about 5 seconds. So, this is what is going to be super helpful for when something goes wrong.

03:39 Um and when something breaks in production, I promise you it will. This is the first place that you want to look. Um you know, there have been countless times where um a customer’s agent has broken and then the trace can show you in less than 10 seconds, um, that the retrieval was fine and it was a problem with the prompt or the other way around. But, you know, it’s not

04:09 uh, everything to be able to see a single trace. Um, you need to know that the agent was good and not that it ran without throwing an error for one instance. Um, so this is where we go to mlflow.genai.evaluate. Um, and this is where we are running a set of scores across a whole bunch of questions at once. So, this is one evaluation run for a set

04:42 of questions and we are having three types of assessments. So, we have relevance to the user’s query, safety, and both of these come at 100%. So, so far that’s healthy. But then, we have this third metric over here, which is, uh, called stays on catalog. Um, and basically this is one that I created in plain English and all I’m asking is, does the agent

05:15 stay on our product catalog or does it wander off topic? And this one is coming back at 80%. Um, and it failed at one question. Um, so this would be the one that’s asking for the capital of France. Um, so you might be thinking, uh, you know, how can we have, uh, this checked as relevance, um, but then saying that the guardrail wondered off? Um, and it’s because they’re measuring

05:45 two different things. So, relevance is checking for whether the answer matches the question. Um and it does. Paris is the perfectly correct answer. Um and safety, was it harmful? Um no. Uh this is not containing any um problematic language. Uh but then if we go on say some catalog, uh we can um see that it correctly flags

06:18 that the question is going off catalog. Um so, even though that the answer was perfectly safe and perfectly relevant. So, this gap over here is not a bug um and that’s the evaluation earning its keep. So, this is the difference between hoping your agent behaves and then knowing it does. And the best part is that the exact same

06:49 scores I’m running here in development uh also run in production. So, um when you’re doing any monitoring, you can set the bar on day one and be able to track that it doesn’t quietly drop the moment you ship. >> [music]