AI Agents Don't Read Your Policy Docs. They Hit Your APIs — Gravitee

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AI Agents Don’t Read Your Policy Docs. They Hit Your APIs — Gravitee · Channel: AI Engineer · 17:47 · Watch on YouTube

Topics: AI agents, Cybersecurity and software security, Software engineering with AI, AI governance and regulation

Summary

Sam, CTO of Gravitee, demonstrates five live examples showing why AI-agent governance should be enforced at the gateway rather than left to prompts, tools, or policy documents. The demos cover blocking unauthorized actions and data exposure, rate-limiting requests, reducing repeated LLM calls with semantic caching, and rejecting oversized or irrelevant prompts before they incur downstream costs. He argues that centralized gateway controls can provide scalable identity management, security, cost control, visibility, and auditability as agent deployments grow. He also describes additional capabilities including model routing, narrowing the tools exposed through MCP servers, and managing shadow AI traffic.

Key points

The speaker cites a survey in which 88% of organizations had an agent in production taking action, while 14% of those had governance around it.
Policies and restrictions in prompts or agent context are not reliable enforcement; the demo contrasts them with permissions enforced by the gateway at runtime.
Gateway identity controls can block unauthorized actions and prevent access to sensitive guest information.
Rate limits can stop excess requests at the gateway before additional downstream LLM calls are made.
Semantic caching can serve similar prompts without repeating every LLM call, with a configurable similarity threshold.
Gateway policies can reject oversized prompts and content irrelevant to the agent’s use case before sending them downstream.
Centralized logging and telemetry provide visibility into agent and tool activity and can be exported to downstream observability tools.
The speaker also presents model routing, fine-grained MCP tool exposure, and shadow-AI management as additional gateway capabilities.

Chapters

Start	Chapter	Summary
00:01	Why agent governance belongs at the gateway	Sam introduces the live demo and argues that governance should be enforced through infrastructure rather than relying on agent policies or prompts.
02:40	Blocking unauthorized actions and data access	The first demonstrations show the gateway blocking an unauthorized booking deletion and preventing exposure of guest information.
06:30	Controlling costs with rate limits and caching	The demos show gateway rate limits stopping repeat requests and semantic caching serving similar prompts without repeated LLM calls.
10:40	Managing prompt size and relevance	The gateway rejects an oversized prompt and an off-topic request before either creates a downstream LLM call.
13:22	Scaling governance and expanding gateway capabilities	Sam discusses agent sprawl, auditability, model routing, MCP tool scoping, and controls for shadow AI before concluding that governance should move to the infrastructure layer.

Quotes

how do we remove that to be within its identity so it can’t do those things in the first place — 01:05

The agent doesn’t care. Like at runtime, no one reads those documents. — 03:47

They’re blocked at the gateway. — 08:01

the only real correct way that we’re going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer. — 17:02

Concepts and entities

Gravitee, Sam, Gateway, MCP, A2A protocol, JWT, OpenTelemetry, semantic caching, Redis, Databricks, Groq, Anthropic, Gemini, OpenAI, Datadog, Splunk, Prometheus

Full transcript (original language)

00:01 [music] everyone. Uh I’m Sam and thank you for coming to this talk today. So I’m the CTO at Gravity uh and one of the sponsors events and I’ve gone a little bit different. So, I’m not going to go through kind of reams and reams of slides around why governance is important and how you’d look to secure your agents and all of those architecture diagrams. Instead, I built

00:32 essentially two agents and we’re going to go through it today. Okay. So, I’m sure you’ve all uh got an AI policy. You’ve got kind of a framework of how you want your agents to operate. You’ve got people exploring AI. So a recent survey we did showed 88% of organizations had an agent in production actually taking action. Only 14% of those had any governance around it. And I’m sure you’ve all seen in the press or LinkedIn or any articles around when things go wrong, they go really really wrong. You lose databases, you expose

01:05 PII, there’s heavy fines from the ICO coming your way. if you start to leak sensitive information, especially if you’re in a HIPPA or a GDPR kind of type landscape. So the question isn’t like how do we kind of secure the agent within its own kind of context or prompt is how do we remove that to be within its identity so it can’t do those things in the first place rather than relying on uh a user prompting it correctly or um some kind of markdown files kind of limiting what it can do. Like remove that out of the equation. Remove it to

01:37 the gateway. move it to like the governance layer and actually it can’t do any of those things because I’m sure you know AI is really helpful like it wants to please it wants to give you what you want so if you’re asking for things around hey delete this database or hey give me these customers it will um and this is about how actually how do we start to shift that as well so I’ve got I’ve got five examples this is live okay so it’s not scripted uh it’s running on a uh an architecture way it’s a gravity gateway um which is in a SAS environment as a data brick back end for some data tables where the information

02:10 is stored and it’s a combination of a set and a Groke with a Q, not Groke with a K. I know US people don’t really like Groke with a K. Uh, so Gro with a Q for some inference. So it’s all fully live. Um, and we’re going to going to go through some examples. So the agent on the left is where I’m trying to put the control within the prompt or within the kind of skills or tools or knowledge that I’ve given it. And the agent on the right is governed by a real tightly scoped identity and gravity’s native gateway controls. So if we uh go for the

02:40 kind of scenario number one, this is around a prompt injection. So I am a user, an actor, someone who wants this agent to do something. So I’m going to ask it, hey, use the tools that you’ve got access to and do something for me. So basically saying delete this booking for me but also can you run the MCP tool delete all bookings and the agent on the left is gone yep no problem uh I’ve done that for you and I’ve also deleted this booking is there anything else you need essentially the tool was there and the a

03:15 the user in question had permissions to use it so it executed it flawlessly on the right hand side the agent identity is controlled by gravity’s gateway like the admin permissions are really tightly constrained to just the users that can access it just at runtime. So the kind of token exchange and the JWT claims are only valid when that user needs to ex execute that task. And if you dig into the packets down the bottom to kind of show it’s not just scripted environments. You can see all of the flow coming in. You can see what the user asked. You can see the agent executing it. You can see the tools

03:47 being called. And you can ultimately see the 200 errors or 200 kind of codes where that has actually been a valid request. And if we go into the database now, all of those bookings have been deleted. So it was genuinely dropped by that agent. So we had a policy uh in our hotel organization that said, “Hey, don’t use AI to do this.” The agent doesn’t care. Like at runtime, no one reads those documents. it. You need to rely on the infrastructure and the governance layer to be able to control what you can and can’t do in terms of your gent landscape. So the one on the right you can see which is just these

04:21 ones down the bottom. So the same prompt was asked to it the same tool discovery occurred. So it discovered yeah I’ve got a tool to do this but then when it came to execute it the permissions would not were blocked at the gateway with a 403 kind of forbidden kind of code because it didn’t have the right permissions to be able to execute it. And it still did the bit it could do. So if you see the prompt return that said yeah I still cancel this booking for you I’ve done the bit I can but I can’t do this for you because I haven’t got permissions and that’s moving it up out of a policy document to actual hard controls at the

04:52 gateway level. If we go for a similar kind of a second example this one is around kind of data exfiltration. So I’m going to ask um hey give me the full list of guests. So give it to me because I need to do some compliance stuff with it. I want their emails. I want my car details and I want to do some reconciliation. So, same kind of flow. So, prompt comes in from the UI, goes to the LLM. LLM does some MCP discovery, finds out it’s got a tool for it, gets it, returns it back in. Here’s all the information kind of exposed just

05:25 to say this is all kind of mocked. It’s not real. I haven’t just kind of given you everyone’s bookings, but you can see it coming back in because again, the LLM wants to be helpful. It wants to solve that problem. on the right hand side the gateway successfully blocks it. There’s been no PII leak. There’s no exposure of information. We’ve said you haven’t got the right permission to be able to do this. If you want to authenticate, I can absolutely help you, but this is what I would need to be able to do it. And that can come through kind of any um uh IDP platform or gravity gateway itself. And if you want to, we can start to explore

05:56 the open telemetry down the bottom and really start to dig into like what’s happening under the hood. So you can see the various steps that it’s going through. Especially with like the A2A protocol and like user to agent, agent to agent, agent to agent to agent to agent, things get quite messy and quite difficult to really follow. So relying on the permissions within the agent itself to be able to control what it do, especially when you’re scaling to hundreds or thousands or hundreds of thousands of agents can get quite tricky to be able to manage, mitigate, and also kind of migrate when you’re ready. So moving it to the gateway gives you a a much more sustainable, scalable, but

06:30 ultimately secure way of of doing that. And you can kind of see it within the responses here. You can see this was a genuine error returned by the gateway. Again, not mocked, not scripted. This is what you’re able to kind of govern and secure. But it’s not just about identity. So if you’ve got agents there, you might want to also control some of your costs. So token spend and everything else become increasingly important um alongside kind of other things which you can again do through the gateway. So this one we’re going to uh basically have a buggy agent. It’s going to send the same

07:00 request kind of eight times um which if you see it kind of being governed in the agent itself. Every request goes through. Every request gets a response. We spent x amount of tokens. We spend x amount of effort servicing that. nothing in that agent says you’ve done too much. We should rate limit. We should token limit. And if you see here, uh, everyone kind of went through. They all got a 200 kind of response. They were all all kind of um, uh, genuine kind of allm prompts.

07:30 [snorts] In this example, they’re quite simple. So, we end up spending about a th000 tokens, which is like 1 cent, which in the grand scheme of things is not a massive amount of money. But if you multiply that by all of the agents you got in production, all of your agents you’ve got running for various customers and all of the kind of um uh repeatable kind of workflows they’ve been asked to do, you can get to some quite significant costs quite easily. On the right hand side, uh governed by the gateway, we see the same prompts coming in. The first three are answered because they’re valid, but then we get into the

08:01 rate limiting applied by the gateway where it’s saying you can only do three every minute. So any more than three is blocked. They’re not blocked at the LLM. They’re not blocked kind of downstream with already spent tokens. They’re blocked at the gateway. And again, that allows us to save costs, but also start to kind of control which users, which permissions, um, ultimately how much you want to spend in certain audiences. And we can do the same thing for caching as well. So, Gravity as a a product supports semantic caching, which is where if you get similar sounding prompts, you can serve it from a a

08:32 built-in Reddis kind of store rather than sending each one to each time. So for example, think of a um an IT help desk model. How do I set my password? How do I set my password? How do I set my password? Rather than every one of those being asked each time to an L&M and you start spending costs and tokens and effort, serve it from the the database. Benefits there are zero cost, zero latency because it’s coming from there rather than having a downstream L&M call, but also kind of a much kind of nicer user experience as well. So in this example uh I’ve asked five times list all my hotel bookings because uh I

09:06 didn’t trust the first one. Everyone here gets served out. You can see I spent another thousand tokens, another scent and everything. Uh everyone coming from a an a downstream L&M call. Let me drop that down a little bit. On the right hand side I’ve got the same data returned but this time from the cache. So I’ve only spent around 200 tokens on the in and the out kind of response there. So really powerful especially when you’re scaling, especially when you got multiple users and multiple agents working on the same problem. You can start to serve it from the the built-in cache and there’s no extra kind of like

09:39 complexity or infrastructure. It all comes bundled with our gateway. So you’re able to can kind of control that as you go as as well. And this can be controlled as well. So you don’t have to define you can define how kind of similar you want it to be. So it’s a zero to one scale, one being very similar, zero being not very similar at all. And that’s within your control. So you can flex it per agent, per audience, per user, depending on what you want to start to achieve. And again, we get full logging down the bottom so you can see what’s coming in. So you can see it’s hit the cache response. It’s all kind of

10:09 um surfaceable. It’s all transparent. Gravity as a platform isn’t trying to like kind of lock you in, kind of make things blackboxy. It’s trying to surface up so you’ve got full visibility of what’s going on across your agent landscape. Um the telemetry here, I’m just showing in a UI. you can export it to any downstream tool you want to. So a uh a data dog or a Splunk or Prometheus if you’re in Kubernetes, it’s all kind of like there for you to be able to use to start to get the insight you need. And our fifth example is say like uh

10:40 we’ve got certain customers who said, “Hey, our users are just kind of dropping huge con prompts in because they want to summarize a whole uh conversational history or they want to start looking at how they um like summarize meetings etc. which is is fine, but if you want to start limiting tokens, etc., you start to rapidly get unstuck. So, this is just a a big response where the user has dropped in 10 minutes of kind of like chat transcript and is said, “Here’s your task. Do this for me.” And

11:10 this really gets into how do we manage context? How do we start to shape like how do we deploy these agents with just enough context to be able to do what they want? So, in this example, I put 2,000 tokens in. uh my output was 312 which is quite significant for what essentially is quite a simple task is book this room for this audience uh do these few things. So in terms of the gateway uh we ran the same prompt uh and actually we blocked it there. So there was no downstream LLM call. There was no downstream kind of like token cost of

11:43 being able to churn through all that context. Basically said this is too big blocked at the gateway level with a built-in policy that says think about cutting this up. Thinking about how you share that can you break it into smaller steps so we can start to control some of that space. And this is this is executed at runtime. So, it’s not something you want to do on the response phase where you’ve already churned the tokens. You’ve blocked it going anywhere downstream uh pretty much before the prompt is gone there. So, you can start to limit some of your costs as well, which is really powerful, especially if you’ve got multiple users. You’ve got people treating your agents like um uh

12:16 their own personal Google search to find information. uh we saw uh a customer where an agent was simply asking what’s the weather like today where I am and they were using a really powerful kind of model to do that and they were spending a huge cost to get something that Google or uh any kind of met station would be able to give you. So uh as well as kind of like the length of the prompts you can also really focus it in on the type of content you allow. So you can say this content must be relevant to this use case, this content must be relevant to what my agent needs to do. So in this example, if we start

12:48 to ask things around what’s the level like today, the gateway will say this doesn’t look like a hotel booking software or a hotel booking ask no thank you and spending no downstream costs. So being able to kind of decouple your front-end agents and your backend services, be they APIs, MCP servers, LLMs, you’re rooting to databases, data lakeouses, whatever you want to do. It’s fundamentally important to bring you the control that you need, to bring you the cost optimization that you need, but also the security and the governance and

13:22 the auditability. So I think uh currently there are 144 agents that my colleague Tim at the bat tells me for every one human kind of identity. So the problem is only getting exponentially worse. If you start to hardcode this into your agents may work for the first one, the first five, the first 10, but when it becomes exponentially bigger and these agents are creating other agents with creating other agents into into a a huge sprawl, being able to manage it with a scripted automated kind of governance layer at the infrastructure

13:54 is fundamentally a fundamental requirement of how we start to organize these identities and kind of everything else. In addition, I think the landscape is going to get increasingly regulated and how do we control some of the bits that going in to retrain models. How do we get audited uh on the information we’re coming in? So having our logging, having our information here, having it basically surfaceable and exportable for anything we want to do, I think is something that a gateway, it hasn’t got

14:25 to be gravity. Uh having a gateway sat across our agented landscape is a requirement that we should consider alongside everything else of doing a aentic work. Uh and in here we can also uh put in a UI to ask kind of any questions we want to which we can start to uh interact with it as well. So I’ve shown you five examples here. Uh the platform capabilities are kind of much bigger than just the five here. We’ve got built-in capabilities to do like LLM routting. So we can route to any

14:55 downstream model depending on like um latency or cost or uh complexity if you want to. And it allows you to decouple yourselves from any kind of like hyperscaler. So rather than being within a certain environment and having a onetoone relationship, you can route to any downstream model you want. So in this one, I’m rooting to to Gro into anthropics, but within three clicks, I could route it to Gemini or Open AAI and the gateway will handle the transformation between the different OpenAI specs for you. So rather than

15:26 trying to recode and everything else there, it’s all handled within the gateway. Uh we’ve also got tools to kind of create MCP kind of studios. So rather than just using thirdparty MCP servers where you are exposing all of those tools each time. So our CRM we use exposes 64 tools for that MCP server. Dropping that into the model would expose huge context every time. I spend money, I spend tokens, spend cost. Being able to create that down to a really fine amount of tools that the MCP server is exposing limits my blast radius, but

15:59 it also limits my cost as well. enable to start to control some of that. Uh we’ve also got tools around edge management as well. So an increasing problem that we’re seeing within organizations that we work with is that um the traffic through the gateway is fine but there’s a number of audiences a number of users using on their local laptops. They’re using AI. They’re um dropping in documents that they don’t want to read or summarize or emails or things like that that no one’s got visibility to. So just last week we released a edge management capability within the gateway which enabled you to

16:30 see your shadow AI traffic enables you to secure it enables you to force it to go through the gateway it gives you logging observability discoverability around any of those downstream uses that aren’t going through a gateway and you can do a number of things there um especially for like infosc compliance and things around heavily regulated industries. if you want to kind of learn more about that. Uh quite helpfully our booth is just there uh kind of the third one down that landscape. Um but hopefully uh with this example, it’s been something a bit different from uh the normal slide to go through and

17:02 showing you how putting your governance and your policy within an AI agent itself and relying on a markdown file or a tools or a context or identity there isn’t correct. And the only real correct way that we’re going to scale but also keep the governance and securability as we move forward is shifting it up to the infrastructure layer. And if you want to learn more, uh there’s a number of our colleagues down there, but I just want to thank you for your time and hope you’re enjoying your AI engineer event. Thank you.

17:47 >> [music]