Governing AI Agents with Unity Catalog and Unity Gateway on Databricks | Unity Catalog Demo

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Governing AI Agents with Unity Catalog and Unity Gateway on Databricks | Unity Catalog Demo · Channel: Databricks · 05:44 · Watch on YouTube

Topics: AI agents, Cybersecurity and software security, Privacy and data protection, RAG, search and information retrieval

Summary

The demo shows how Unity Catalog and an AI Gateway govern an agent’s access to data and the answers it returns. A notebook sets up built-in safety policies, custom Unity Catalog functions, and role-based filters that use user groups and data tags. Alice, in support, and Bob, in finance, ask the same question about Acme, but the filtering gives each a different set of relevant context. The example illustrates how filtering data before it reaches the agent can produce user-specific results.

Key points

A notebook configures the Unity Catalog governance controls used in the demo.
The AI Gateway includes built-in policies for jailbreaks, unsafe content, and sensitive data.
Custom policies use Unity Catalog-managed functions to check requests and block specific export operations.
A role filter uses Unity Catalog tags and user groups to determine which data users can access.
Both Alice and Bob can access public data; Alice can also see support data, while Bob can see finance data.
The demo uses customer and knowledge-document tables, including records for Acme and Globex Retail.
The notebook emulates vector search to find documents relevant to a question about Acme.
The same question produces support-focused answers for Alice and finance-focused answers for Bob.
Unity Catalog filters data before it reaches the agent, providing context relevant to the user.

Chapters

Start	Chapter	Summary
00:00	Notebook and governance setup	The demo introduces Alice and Bob and shows the notebook used to configure Unity Catalog governance.
01:03	Gateway policies and role filters	The presenter reviews built-in and custom policies, then explains how user groups and tags control access to data.
02:38	Demo data and context retrieval	The video shows the customer and knowledge-document tables and explains how the notebook finds context for a question.
03:39	Comparing results for Alice and Bob	The same question about Acme is queried through the gateway, returning support information for Alice and finance information for Bob.
05:12	User-specific agent results	The presenter concludes that filtering data before it reaches the agent provides different, user-relevant results.

Quotes

This notebook sets up all of the Unity Catalog required governance controls — 00:31

we already ship with jailbreak, unsafe content, content, and sensitive data policies. — 01:03

Alice is in support, so she can see support information and Bob can see finance. — 04:40

Unity Catalog filtering the data before it hits the agent — 05:44

Concepts and entities

Databricks, Unity Catalog, Unity Gateway, AI Gateway, UDF, busca vetorial, Alice, Bob

Full transcript (original language)

00:00 [music] >> Let’s look at this from a from an example. Uh you remember Alice and Bob. Alice and Bob are two different users seeing two different results when querying a model because it’s governed by Unity Catalog. What we have built here is a notebook that implements all of the example information that we just discussed. Don’t really get hung up on the details. This is a typical data

00:31 engineering task. We created a notebook. We encoded all of our rules policies in this notebook and we can see the data that we have created. The important part is that this notebook sets up all of the Unity Catalog required governance controls and I’m going to show you these directly in the Explorer. So here in the Explorer you can see the catalog called governance demo and inside we set up an AI gateway. That AI gateway has been configured with

01:03 a set of policies. Three of these policies are building ones. So we already ship with jailbreak, unsafe content, content, and sensitive data policies. But we also created two custom one that wire into UDFs that are managed by Unity Catalog. We have one that checks the model before a question is asked and we block specific export operations. You can see those functions here as well. So ask before external actions is here. Really just a function that checks

01:36 the input and then returns a response that the agent can follow. The same for block bulk customer export. We are getting some input input input here in form of a parameters and then decide if this is right or not and then return the results that the AI gateway and the agent can react to. We also set up a a role filter. That role filter use a specific tag in Unity Catalog and the groups that

02:07 the users are in to decide what roles they can see. Remember that I said that we have Bob and Alice. Bob is in finance and Alice is in support. So, you can see that both of them can see uh public uh data sets, but they can also public data. They can also see either support data if they are in the support group or they can see finance information if they are in the finance group. So, either or, but both can see public. Remember this, we will see this in a second also when we run the notebook.

02:38 So, all of this is being set up here and we have two tables. One is a customer fact table which holds some information that we um have stored for a for a set of demo customers uh and those customers here have really just uh or we have just two customers here. Uh Acme and the Here we are. Acme and Globex Retail. So, we have two customers um that we can look at. Remember this again, we will ask for Acme later on as in our example, so we will don’t want to see anything from

03:08 Globex. We also have a set of knowledge dogs that we can look at. This will build our um this will build our context. So, based on the question that you ask, uh the query that we will run is based on which company we are asking for and what scope you can see like public or finance or support and the information that you can get out of the document. Back to the notebook. So, if I switch back to the notebook, here it is.

03:39 We set everything up and then you can see we are running a query here across uh a lookup. This is basically a vector search, but it’s emulated in this notebook um for simply uh simple uh yeah, for reasons of keeping it simple, um but you can see I can as a manager or I can see here as the main owner all of the records. That is wrong for Alice and Bob. They can only see subsets of this. So, how does that work? We’re setting up more details in here. We’re setting up

04:09 the AI gateway, which has the incoming and outgoing UDFs. We have the role filter. So, let’s have a look at the question that we’re asking. So, in this example where I have all of the data sets, this is the question that we’re asking for. The lookup is really into the vector search, finding all documents related to this question. Important is we’re asking for Acme and we’re asking for renewal risk or open and open support issues. Now, down here in the

04:40 per user query, we’re using now a there’s the AI model and then the AI gateway and ask the same question against that gateway both as Alice and as Bob. You remember that Bob can see both can see public. Here’s Alice with public and here’s Bob with public. And Alice is in support, so she can see support information and Bob can see finance. With the same data sets, the same input, the same two sets of data files that we have and tables that we have, you can

05:12 see that the results the agent returns is different. For Alice, we get answers back that pertain to the support domain. You can see that it talks about what can be done here, for example, the the follow-up on the um the run book as we discussed in the example and other details. You can also see for Bob down here that his answers are all about finance. So, this shows you that with the same input

05:44 but different filters based on the provenance based on the user running the query. We see see two different results. That is the power of Unity Catalog filtering the data before it hits the agent giving the agent the context that is that is important for the user asking the question either Bob or Alice or me and we get different results all together. So that is important and solves this problem holistically. >> [music]