Building an Agentic Analytics Foundation on the Lakehouse

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Building an Agentic Analytics Foundation on the Lakehouse · Channel: Databricks · 12:22 · Watch on YouTube

Topics: AI agents, Data, databases and embeddings, Privacy and data protection

Summary

The video presents an agentic analytics foundation on Databricks designed to help AI agents answer questions accurately, securely, and at scale. It explains how curated data, the Genie ontology’s defined and learned semantics, Unity Catalog, and Unity Gateway work together to provide context and governed access. A Brick Commerce demonstration shows how Genie agents, Genie One, third-party agents connected through MCP, and governed LLM access use this foundation. The video concludes by mapping accuracy, access, and scale to the platform components that address them.

Key points

Agentic analytics moves from dashboards read by people to agents that can reason, query, and act on data.
The key risks of deploying agents on enterprise data are inaccurate answers, inadequate access controls, and performance challenges at scale.
The foundation combines context, connectivity, and compute.
LakeFlow moves data through a medallion architecture to curated gold-layer datasets, which are governed through Unity Catalog.
The Genie ontology combines explicitly defined Unity Catalog Semantics with learned knowledge from organizational data usage.
Domains and subdomains organize data assets; pages define business concepts; metric views specify measures, fields, joins, and filters.
Genie agents answer questions within a domain, while Genie One can reason across domains and consult trusted organizational assets.
The Genie One MCP lets third-party tools use the same ontology context.
Unity Gateway governs LLM access with policies, rate limits, usage tracking, and guardrails.
The video associates ontology with accuracy, Unity Catalog and Unity Gateway with governed access, and managed compute with scale.

Chapters

Start	Chapter	Summary
00:00	Agentic analytics and its risks	The speaker introduces agentic analytics and the accuracy, access, and scale risks agents face without a suitable foundation.
03:05	Brick Commerce data and business semantics	The demo follows curated sales data into Unity Catalog and shows how domains, pages, and metric views provide business context and definitions.
06:42	Learned knowledge and Genie experiences	The video explains how learned ontology snippets complement explicitly defined semantics, then demonstrates domain-scoped Genie agents and cross-domain Genie One.
09:18	Third-party access and Unity Gateway governance	The demonstration connects a third-party tool through MCP and shows Unity Gateway policies and guardrails governing model requests.
11:20	Putting the foundation together	The speaker recaps how the ontology, governance, and managed compute address accuracy, access, and scale, then invites viewers to build their own foundation.

Quotes

without the right foundation, those agents just produce faster, more confident, wrong answers. — 00:00

The first is context, giving agents the knowledge that they need to answer questions accurately. — 01:34

You could think about them as a governed wiki for your business concepts. — 05:07

the Unity Gateway is the single pane of glass to govern access to your large language models. — 11:20

Concepts and entities

Databricks, agentic analytics, Genie ontology, Unity Catalog, Unity Gateway, Genie One, Genie agents, MCP, Metric Views, OntoRank, Brick Commerce, Justin Kolb Peck

Full transcript (original language)

00:00 [music] >> Every organization wants AI agents that reason over their data. But without the right foundation, those agents just produce faster, more confident, wrong answers. In this video, I’ll show you how to build a robust agentic analytics foundation on the Databricks platform that ensures your agents are accurate, secure, and ready to scale. Hi, my name is Justin Kolb Peck. I’m a senior solutions architect here at Databricks, and let’s get into it. So, what is agentic analytics? Agentic analytics is the next evolution of the

00:32 modern data stack. We’re moving from dashboards that humans read to agents that can reason, query, and act, and semantics are absolutely pivotal in making that shift work. If you’ve worked in data, you’ve seen this problem before. Data teams are buried, and every question that anyone in the organization needs an answer to becomes a new ticket. Thankfully, agents can break that bottleneck, but only if the foundation is right. Because when you turn agents loose on enterprise data without the proper guardrails, you’re exposing yourself and your organization to three

01:03 real risks. The first is accuracy. Agents don’t know that revenue means net revenue in your organization or that active customer has a very specific definition. So, they write SQL that looks right, but gives you the wrong answer. The second is access. Agents often operate in silos because they don’t have the right secure, governed access to the appropriate systems at the right time. And lastly, scale. More queries from more human and agentic users can cause performance issues at larger scale if

01:34 not managed properly. At Databricks, we solve for that with our agentic analytics foundation, and we address these problems in three separate areas. The first is context, giving agents the knowledge that they need to answer questions accurately. That’s learned behavior from how your organization uses data, plus modeled semantics where you formally define your metrics and your business concepts. Databricks makes these available through Genie ontology. The second is connectivity. Securely connect to agents wherever the

02:04 work is happening, whether it’s through our AI co-worker, Genie 1, custom agents that you’ve built with Agent bricks, or any other third-party agent that connect via MCP, for example, Claude code. And lastly, compute. Lakehouse compute is the underlying compute to run analytical queries in real time with lowest TCO possible, and it’s built for scale. Here’s how it all comes together. It starts with the data foundation. Data is ingested and transformed using Lake flow until you have clean, curated gold layer data sets.

02:35 Those data sets then get enriched with semantics through the Genie ontology. These are things like metric and business concept definitions, as well as learned knowledge about how your organization actually uses its data. All of this is governed through Unity Catalog. From there, agents don’t hit this data directly. They go through the Unity Gateway, which governs access to models, MCPs, and more through budget policies, access controls, and guardrails. With that foundation in place, you get accurate, secure AI access through four

03:05 primary surface areas. Genie 1, which is your AI co-worker that reasons across your entire data estate, coding agents, which you can connect to via MCP servers, custom or Genie agents that you’ve built for a specific domain, and Genie code, which is our built-in coding assistant for engineers. So, that’s the architecture. Now, let’s see how this works in practice. Before we get into the demo, let me first introduce you to Brick Commerce. Brick Commerce is a fictional e-commerce company that we’ve created for the purposes of this demonstration.

03:35 We’ll showcase how both human users and agents can leverage Brick Commerce’s enterprise data using this agentic analytics foundation that we just described. Okay, so here we are in the Databricks UI, and we’re going to start where every analytics foundation starts, with the data. In this case, Brick Commerce has data coming in from transactional systems, payment processors, and other systems. LakeFlow takes all that data, moves it through a medallion architecture, so you have your bronze, your silver, and finally your gold layer data sets that are curated and ready for analysis.

04:06 Here, we’re looking at the gold layer assets specifically for the sales domain. So, fact sales, this is our gold level fact table, and here we can see some of the sample data that we’ll use for the rest of the demo. Having clean data is absolutely necessary, but it’s not sufficient. If an agent or even a human sees hundreds of tables with cryptic names, they’re not going to have any idea where to start their analysis. That’s why we organize data assets using domains and subdomains in Unity Catalog. To do that, we’ll head over to the Genie

04:36 1 UI. We’ll click on the discover tab. And from here, I can search for all the domains that might be relevant for my use case. I’ll type in Brick Commerce. And here we’ll see the Brick Commerce domain with all of these subdomains that map to business concepts, like sales, inventory, payments, and credit. We’ll take a closer look at the sales subdomain. And from here, we can see all the different data assets that are related to sales. For example, we have pages, Genie agents, metric views, and other assets as well.

05:07 Let’s drill into each of these assets, starting with pages. Knowing where to look isn’t the same as knowing what things mean, and that’s where pages come in. You could think about them as a governed wiki for your business concepts. Here’s a page called Forge Desk. Now, Forge Desk is a specific sales channel that operates within Brick Commerce, and this page provides the authoritative definition for what Forge Desk is. On this page, you can see we’ve defined an owner, synonyms, a description, but most importantly, a precise definition for Forge Desk. In this case, Forge Desk

05:39 is Brick Commerce’s direct sales channel, and in the data, it’s defined by this specific filter. This is the kind of context that makes or breaks agent accuracy. Once we define what our concepts mean in terms of the business context, we also need to define how they’re calculated. That’s where metric views come in. Metric views are your source of truth for defining measures, fields, joins, and filters. Let’s take a look at our example here and the sales. Over here, we can see all of the measures and all the fields we’ve defined. So, when agent comes and asks,

06:10 “What is the approval rate percentage?” It doesn’t need to improvise. It’ll go here and reference the precise definition for how we calculate that based on the expression provided right here. All right. Let’s review where we’re at so far. So, domains and subdomains tell agents where to find assets organized around business terminology like sales. Pages tell agents what your business concepts mean, and metric views tell agents how to calculate them. Together, these form what we call the Unity Catalog Semantics portion of the Genie Ontology. These are the things that you explicitly define and curate.

06:42 But, there’s a second side to the Genie Ontology, inferred or learned knowledge. In reality, not everything can be hand-curated upfront. Your organization has hundreds of implicit patterns in how people actually use the data. Genie learns from these usage patterns to curate a map of what we call snippets. Snippets include things like metric definitions, authoritative sources, business rules. Let’s show an example here. Here, we’re looking at a metric definition that Genie Ontology inferred, and it clarifies how seller performance metrics should be categorized. So, the

07:13 Genie Ontology is really two halves working together. Unity Catalog Semantics, the things that you explicitly define, and learned knowledge, what the system picks up from how your organization actually works with data. Now that we have a strong semantic foundation with Genie Ontology, everything we’re about to show builds further on this. All right. Now, let’s see this in action. We’ll start with Genie agents. Genie agents are conversational interfaces that are scoped to a single domain. In this case, we’ll look at Brick Commerce Sales, and this one is tuned specifically for sales. It knows the tables, the metrics, the business

07:44 context. So, we’ll go ahead and ask, “What is the monthly trend of approved GMV?” GMV might not be obvious, but the Genie agent has full context on what gross merchandise value is and the specific metric view for how to calculate it. Wait for our answer. Great, we got our answer here. I can trace this back to the source and see that it used the metric view for sales. And it used this SQL expression to define GMV. So, fast, focused, and accurate within the scope of sales. But,

08:16 what about questions that cut across domains? That’s where Genie One comes in. Genie One routes to the right Genie agent, but also considers other trusted assets across the organization, things like dashboards or metric views. It uses the full Genie ontology, both Unity catalog defined semantics and learned knowledge. So, to show this, we’ll head over to create a new chat. I’ll ask a question like, “Please analyze monthly sales in ForgeDesk.” Now, this can’t be answered just by using one Genie agent. Genie One needs to understand what sales means, but it

08:48 also needs to understand what ForgeDesk actually is. We can see how it traces through the reasoning. It’s going to consult the ontology using our OntoRank algorithm to weigh the authority of each asset. For example, a certified metric view will outrank an ad hoc notebook. Okay, great. Here’s the result. We can see the answer, of course, but more importantly, we can see where it got its answer from. Searching the Genie ontology, we can see that it referenced that ForgeDesk page we looked at before, as well as this ontology snippet that it inferred from the data.

09:18 So, every number traces back to a governed definition and we’re able to cut across domains by using Genie One. Now, maybe I run my day-to-day from Cloud Desktop, but I still want all that ontology context. That’s where the Genie One MCP comes in. You can connect to Genie One in the Genie ontology from your favorite third-party tools. Take a look, we’ll go over to customize. Connectors, I can see I’ve registered the Genie 1 MCP from Cloud Desktop here. Now, I’ll go ahead and ask the same question that we asked before. Analyze monthly sales trends in Forge

09:48 Task. We can see that it’s leveraging the same ontology assets as before and it ultimately arrives at the same answer. So, whether you’re consuming through the Genie 1 UI or a third-party agent like Cloud Desktop, you’re able to leverage the same semantic foundation. Agents also need governed access to LLMs. In this case, I’m connecting to a Claude model registered in the Unity Gateway, which provides model governance, cost observability, and guardrails. That’s what UG Claude here represents. Now, over here we can see how I’ve registered Claude to point to the Unity Gateway. Now, whenever I ask a

10:19 question, the request is going to be governed by the rate limits, policies, and guardrails that have been set in the Gateway. I’ll go ahead and ask a question like, “Who are our top 10 account executives?” In this case, it just so happens that our organization has a policy that restricts users from asking questions like this, where it’s inquiring about the performance of other individual contributors. And as a result, the request is blocked. Let’s take a look to see how this is implemented on the Gateway side. So, for that, we’ll head back to the Databricks UI. We’ll click on Unity Gateway here, which

10:50 will bring us to our model service. This model service has been configured to provide large language model access to Cloud Desktop, like we just saw. Over here, we can define usage tracking, inference tables for logging requests and responses, rate limits, and of course, policies, like we just saw in action. Here, we’ll take a look at our allowed topics policy, which defines which topics are allowed and which topics are off-limits. Here, we can see that anything about individual contributors metrics are going to be flagged. In addition to the custom guardrail we

11:20 saw, Databricks Unity Gateway has built-in guardrails for sensitive data detection, unsafe content, jailbreak, and hallucination. In short, the Unity Gateway is the single pane of glass to govern access to your large language models. Okay, so let’s bring this all together. We’ve walked through an end-to-end example of how you can build an agentic analytics foundation on Databricks. We started by ingesting data into the lakehouse and defining rich semantic context with the Genie ontology. We saw how that ontology powers accurate answers across multiple surface areas.

11:51 Genie agents for scope domains, Genie 1 for cross-domain reasoning, and coding agents through MCP. And we showed how the Unity Gateway keeps everything governed and secure with guardrails and policies. Remember those three risks that we started with? Accuracy, solved by the Genie ontology, which gives agents the semantic context to write correct queries grounded in authoritative definitions. Access, solved by Unity Catalog governance and Unity Gateway guardrails, ensuring agents only surface information that they should.

12:22 And finally, scale, solved by Databricks managed compute layer, giving agents access to real-time data with cost controls built in. So, here’s your call to action. Try building your own agentic analytics foundation using the Databricks platform. We’ve linked all the documentation and getting started guides you need in the description below. Thanks for watching.