Middleware for Managed Deep Agents

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Middleware for Managed Deep Agents · Channel: LangChain · 04:13 · Watch on YouTube

Topics: AI agents, Software engineering with AI, AI infrastructure and hardware, Privacy and data protection

Summary

Middleware extends an agent’s life cycle with customized behavior for policy enforcement, fault tolerance, and other needs. The video demonstrates built-in PII middleware that redacts email addresses before input reaches the language model, preventing the agent from using the email for a lookup. It also shows custom middleware that wraps tool calls to log which tools are used. The speaker recommends starting with pre-built middleware and consulting the documentation when building custom functionality.

Key points

Middleware adds customized behavior to an agent’s life cycle, including policy enforcement and fault tolerance.
Potential uses include logging and analytics, prompt transformation, retries, error handling, rate limits, and guardrails.
Built-in PII middleware can detect and redact email addresses in input before it reaches the language model.
Redacting an email can prevent an agent from completing a lookup that depends on that email.
The middleware graph shows where middleware runs in the agent’s process.
The speaker recommends trying pre-built middleware before creating custom middleware.
Custom middleware can wrap tool calls to log which tools are being used.
Tool-call logs can be sent to tracing or used to track performance.

Chapters

Start	Chapter	Summary
00:00	Middleware overview and use cases	The speaker introduces middleware as a way to customize an agent’s life cycle and lists common applications.
01:05	Redacting email with built-in PII middleware	The demo adds PII middleware to redact email addresses from input before it reaches the model.
02:40	Creating custom logging middleware	The speaker creates custom middleware that wraps tool calls so their use can be logged.
03:42	Viewing logs and next steps	The demo shows tool-call logs and closes with a recommendation to explore the middleware documentation and start with pre-built options.

Quotes

Middleware is a way to extend your agent life cycle with customized behavior. — 00:00

Some potential use cases include tracking with logging or analytics, transforming prompts, adding retries or error handling logic, or adding rate limits and guardrails. — 00:33

The agent will attempt to do a lookup, but it actually won’t work because it doesn’t get the email. — 02:07

In this case, this is going to come around every single tool use because we want to log what tool is being used. — 03:10

In general though, the pre-built middleware is a great place to start. — 04:13

Concepts and entities

middleware, PII middleware, Managed Deep Agents, LangChain, LangSmith, Nathan

Full transcript (original language)

00:00 I’m Nathan. I’m a PM here at LangChain and today I’m going to be talking about middleware for managed deep agents. Middleware is a way to extend your agent life cycle with customized behavior. It allows you to manage things like policy enforcement, fault tolerance, and more when your agent interacts with tools, interacts with the LM, or hits limits like token usage. We have great documentation about how to get started with middleware and our middleware SDKs all play great with managed deep agents. So, you can take principles you may already know or use principles that already exist out in the wild to build

00:33 great middleware for your managed deep agents. Some potential use cases include tracking with logging or analytics, transforming prompts, adding retries or error handling logic, or adding rate limits and guardrails. We’ll add a few examples to a managed deep agent I’ve been working on. Let’s get started. We’ll add a PI middleware to redact email. In this agent, I have a tool which allows look up by customer email and we’re going to break that with the middleware, but to start let’s show the current behavior. If I ask what plan is

01:05 Ops on, then the agent should be able to use that look up by email tool to fetch the customer data and answer the question specifically. So, it found the email, it ran the tool call using that email. If I look at the line graph, you’ll see it’s got the middleware set up and it passes things to the model and in this case there’s no middleware yet, but we’ll add some and we’ll see this graph expand. So, to add the PI middleware, I’m just going to add into this middleware attribute here.

01:37 In this case, I’m setting up PI middleware with the attribute email to be blocked. I want to use a redaction strategy and I do want to apply this to the input as it passes through to the LM. If I look back here, I’ll see the middleware added to the graph. If I go back to the chat and I try asking the same question, but this time using the middleware, we’ll see what plan is the email on. And when I send this request in, this pre-built middleware will automatically detect the email and

02:07 redact it and prevent the agent from seeing it. The agent will attempt to do a lookup, but it actually won’t work because it doesn’t get the email. So, if I look back here, you’ll see what plan is redacted email on. So, this isn’t even stored inside of LangSmith. And if you look at the graph, you can see that this is passing through. It runs the PII middleware and then it redacts it at that step in the process. So, before it even hits the model, the input has been modified in place. If you’re trying to block specific types of PII, you could use that PII middleware. Other tools in here are

02:40 really handy for common cases. So, definitely experiment with different options here depending on what you’re trying to do. And it’s always worth using the pre-built middleware before you go ahead and build your own. But in some cases, you’re going to want to define custom middleware. In this example, we’ll create some custom middleware that just does something basic, logging. So, I’ll create a new middleware directory in my project and I’ll add this audit file. And in this audit file, I’m just going to bring in the example here. And this uses this wrap tool

03:10 called decorator, which is going to automatically wrap any tool call that’s used. And there’s a bunch of different kits from the middleware package that you can use depending on where you want to place your middleware in the agent life cycle. In this case, this is going to come around every single tool use because we want to log what tool is being used. Okay, so I’ve set up the middleware and now I’m going to add this to my agent definition. So, if we take a look at the line graph here, we can see we have this patch tool

03:42 calls middleware. And this runs at the beginning. And if I interact with this agent, for example, sending the same question that we’ve been asking already, and I take a look at the agent logs. When I run this, we’ll see this is going to actually log the tool calls that are being used. Here, it’s just logging them into my standard out, but obviously you could send this to tracing, you could track performance, or whatever you need to do in this case. So, here’s the tool calls that were run. So,

04:13 that’s a quick introduction to middleware and using both the built-in middleware and the custom middleware functionality. We have a ton of great documentation on building custom middleware, so I definitely encourage checking that out to see more if you’re trying to get more creative with how you build. In general though, the pre-built middleware is a great place to start. So, definitely give it a look if you’re curious about adding more custom functionality around your tool calls or controlling the set of information that’s sent to the LLM.