Three Questions Every Agent Engineer Should Be Able To Answer

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Three Questions Every Agent Engineer Should Be Able To Answer · Channel: LangChain · 14:00 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models

Summary

The speaker frames “owning your own intelligence” as the ability to understand agent failures, reproduce them reliably, and make changes that prevent them from recurring. Production traces can reveal failure patterns at scale, while evaluation environments that model the agent’s real-world context make those failures more reproducible than simple input-output tests. Teams can turn production telemetry into challenging tasks and benchmarks, then improve the agent through changes to its harness or model. Robust evaluations also help teams adapt as models and user interaction patterns evolve.

Key points

Agent builders should be able to identify what went wrong, reproduce failures, and determine how to prevent them in the future.
Mining production traces can reveal useful signals, including incorrect tool calls, missing context, and frustrating user experiences.
Evaluation-specific judges can help classify agent behavior at scale, where relying only on frontier LLM judges may be expensive.
Reliable evaluations should model the agent’s broader environment, including database state, permissions, schemas, and past actions.
World specifications define stable parts of the environment, while tasks test specific actions and possible failure modes.
Tasks should be genuinely challenging and designed to avoid reward hacking.
Application teams can use their own production telemetry and knowledge of real user interactions to create tasks and environments tailored to their agents.
Once a benchmark is established, teams can improve the agent through harness changes, such as prompts and tool schemas, or through model-layer changes.
Robust environments and tasks help detect regressions and adapt as models absorb harness capabilities or user interaction patterns change.

Chapters

Start	Chapter	Summary
00:03	Three questions for agent builders	The speaker introduces the questions of identifying agent failures, reproducing them, and preventing them from happening again.
01:40	Finding failure signals in production traces	Production traces and evaluation-specific judges can help surface, classify, and cluster agent failures at scale.
03:50	Reproducing failures with realistic environments	The speaker explains why input-output tests are insufficient and describes modeling an agent’s world and tasks with specifications.
08:41	Turning telemetry into tasks and benchmarks	Application teams can use their production data and knowledge of real user behavior to build challenging evaluation tasks and a foundation for quantitative improvement.
11:21	Improving agents and adapting over time	Teams can change the harness or model, rerun evaluations, and use robust tests to track regressions as models and user behavior evolve.

Quotes

there are three main questions that any person who’s building an agent needs to have a very solid grasp of. — 00:35

There’s a variety of different signal within your agent traces, particularly your production traces — 02:12

through the idea of working with specifications to define the agent’s world as well as the tasks within it. — 05:30

Really then changes come at two layers. — 11:21

being able to adapt to changes as models improve, as user interaction patterns change. — 13:29

Concepts and entities

LangChain, production traces, LLM as a judge, world spec, GTM agent, Gong, Salesforce, Calendly, Google Calendar, reward hacking

Full transcript (original language)

00:03 Welcome, everybody. Really looking forward to covering how and why and the methodology around what we know at LangChain to be about owning your own intelligence. It’s our new office. Really stoked to have everyone here. We’ve got an awesome lineup. I’m going to approach what it means to own your own intelligence from a practitioner’s angle. For someone who’s building with agents and for someone who is probably going through a lot of the processes that it takes that I’m about to cover when it comes to

00:35 building and improving your agent. I think that there are three main questions that any person who’s building an agent needs to have a very solid grasp of. And if they can answer them well, then they can definitively say that they do own their own intelligence. So, that starts with when you’re building agents, understanding what goes wrong and understanding where your agent fails. This is obviously a very, you know, this is a simple question, but actually there’s a lot of engineering work that goes into understanding if an agent failed. There’s also a lot of engineering understanding that goes into understanding how we

01:07 can make it fail again. This is obviously key on the process to being able to build agents that improve and to be able to own all parts of your model, and maybe the weights, and the all your harness and the weights, and then the weights as well. And then finally as well, being able to answer what needs to happen to make it stop failing in the future. I think to answer the first question and to understand really how and why your agent has failed requires a very strong grasp of tracing and understanding your

01:40 agent’s behavior. And this often comes from mining your agent’s production traces. Building an agent, hopefully you’re tracing it, you’re going to have a lot of trajectories and understandings of all of the different kinds of behavior that your agent is taking. And amongst all of this, there’s going to be a lot of valuable signal that you can use to move on the path to improving your agent. That might be, for example, times where the agent either successfully called a tool or it incorrectly called a tool, it led to frustrating the user because it

02:12 didn’t have the right context. There’s a variety of different signal within your agent traces, particularly your production traces, that is a very important element to capture when it comes to owning your own intelligence and understanding whether or not your agent failed. The issue with doing that at scale, obviously, is that if you have an agent that has thousands of runs a day, millions of runs a day, it’s going to be very hard to understand at scale whether or not your agent is performing as you expect it to. Using frontier judges, LLM-as-judges, online evaluators, obviously is very

02:46 effective, but also very expensive at scale. And so, as an example, a way that we approach this, and just one of the ways that you can look towards understanding and extracting signal from your agent traces, is by making the evaluation-specific judges to go and extract and actually mine your agent for production signal. So, we did this at LangChain recently. We worked with fine-tuning, labeling, and producing a judge to go and actually classify whether or not an agent was giving users a delightful experience, whether

03:17 or not there was any perceived error. So, that might be, for example, an agent frustrating user, or it might be a, you know, a string of tool calls that an agent makes that actually leads it to performing the wrong action when a user was looking for it to do something else. And so, being able to actually have some form of monitoring and understanding what your agent’s doing at scale is a very important step towards understanding whether or not it failed. And also a very important way to be able to cluster the behaviors and the failure mechanisms that begin to emerge as you build and as you start to improve your agent.

03:50 This is only the first step in the journey, though, because once we know and once we have classified agent behaviors, we then need to go and take a variety of next steps to go and be able to reproduce and then actually remediate these issues as they continue to develop. So, that brings me to my next point about what it really means to be able to own your own intelligence. A very important part of that is being able to reliably reproduce where your agent is failing. And I’ve worked with a variety of different application teams, a variety

04:23 of different teams building their agents, and really over the past six or seven months, this has begun to take a pretty drastic shift. The typical way of doing this, and actually, rather, the shallow way of looking at this, is just evaluating whether or not the inputs, or evaluating whether or not the outputs to your agent given some simple input produces the behavior that you’ll expect. This is very powerful for capturing a variety of different agent failure mechanisms, but often doesn’t encapsulate the full picture. The reason why is an agent’s behavior might be influenced by external factors of

04:56 the environment that it’s in. And so, that might be, for example, the state of the database that it’s interacting with, the permissions that it has access to, the particular schemas that it’s working with, or even the past actions that it’s taken that has led to it being in the environment in the state that it has. And so, what we really need then is a way for us to be able to mock out the broader application machinery of your agent as it stands in production. And that brings me to building more robust evaluations by creating a way where we

05:30 can mimic the world or the environment that your agent exists in. This is an iterative process, but really what we… a way that we found to do this quite successfully at LangChain, and again, this is something that’s constantly evaluating, is through the idea of working with specifications to define the agent’s world as well as the tasks within it. So, what I mean by that is having some kind of generic world spec which defines the things that don’t mutate across that agent’s world. And so, that might be things like the permissions, or that might be like the

06:03 domain-specific rules that an agent has access to, or that might be things like the state of the file system and the dependencies that it interacts with. This is things that don’t change across the agent’s actions that it takes in this action space. So, an example of this is we have a very commonly used internal agent at LangChain. It’s called the GTM agent. And so, our go-to-market organization will use it, for example, to do things like do competitive research on competitors in deals, or go and understand

06:34 and spec out the pipeline of a particular sort of opportunity. The world in that case would be all of the systems that the agent needs to interact with: Gong, Salesforce, Calendly, you know, Google Calendar. And effectively the systems that it touches and interacts with as this agent interacts and is engaged with by all of our sellers. Tasks within that world represent specific actions or tasks that the agent needs to take, which will go and actually test and hopefully surface different

07:07 emergent behaviors and things you’re trying to test for. And so, a task within that go-to-market agent world, for example, would be go and research all of the Gong transcripts for a particular customer that we’re looking to investigate or are in the process of working with, and help me draft a next best reach-out. And so, if I’m somebody who’s building a… or if I’m somebody who’s building an environment or a place where I can reliably test my go-to-market agent in production, I’m probably going to want to have some concept of

07:37 a robust world where it mimics the environment that that agent is operating within production. And that extends far beyond just a simple input and output based evaluation. One thing that we’ve noticed, though, when kind of going and creating these worlds and developing these specs and working with an agent, coding agent, side by side to actually go and develop these, is that agents themselves aren’t very good, or rather, need actual constant supervision and guidance when building these out. And so something that’s obviously very important in all of this is to make sure that the tasks that your agent is defining are both reliably

08:09 hard and actually present a challenge for you to be able to continuously improve your agent with. And also there aren’t methods where the agent has designed tasks that are easy to reward hack. You might have heard of the term reward hacking or, you know, adjacent terms. This is obviously really important as well because it makes sure that the world that we’re developing and the tasks that we’re giving our agent are sufficiently hard so that when we make improvements they’re quantifiable and actually mimic really what the agent would be doing in production.

08:41 Something that I think is kind of really interesting and core to this, and once we have this kind of like mechanism now of knowing that the way for us to really kind of get a handle on our own intelligence and be able to create and develop an agent that is solving problems in a world that actually mimics the world that it’s, that the kind of environment that it’s operating in, is that we have the ability to go and do this through turning that data that we have from running our agent in production into this kind of, these kinds of tasks, into these kinds of worlds.

09:15 So it’s no surprise that some of the best models that we use today, produced by frontier labs, are… it’s no surprise that frontier labs are producing some of these models. And the reason why, obviously there are many reasons, but the capital that they can allocate towards acquiring data to build these worlds and to build these tasks far supersedes that of the average app-layer company. A classic example of this, by the way, is, you know, Google making a $10 million tender offer to Spirit Airlines not to buy the company, but to buy the internal

09:45 Slack memos, the emails, the documentation, because obviously all of that’s going to be used to develop tasks and develop you know, training datasets to be able to actually go in and produce and make better models. But this is a bit of a David and Goliath situation because, unlike frontier labs, application developers and application owners see the spiky edges around the ways that people are using their agents. And for that reason, they’re able to produce data from that telemetry, or they’re able to understand how real people are actually interacting with and using their

10:18 agent, and therefore develop tasks and develop environments that a frontier lab never could. And so, once we know that from the traces of our agent, from all of this exhaust, from all of this substrate, we can really help design powerful tasks and worlds to be able to go in and actually help improve our agent. Then obviously we know that it’s a data-wrangling problem, because there’s going to need to be deduplication, there’s going to need to be all sorts of different ways that we turn these traces into reliable tasks.

10:50 But once we do, we have the substrate and we have the foundation for a lot of further agent improvement and development, and really then we have a strong benchmark and starting point for us to be able to go and actually make quantitative agent improvement. And so, once we have a world, once we have a set of tasks within that world, ideally a task targets some kind of failure mode that you’ve seen in your agent, right? You would design tasks around where you’ve observed your production

11:21 traffic, or you’ve observed your production agent to be failing. Really then changes come at two layers. And this is really where owning your own intelligence and owning the mechanisms around both the harness and the models that are powering your agents really begins to compound, because now you can have full flexibility to make changes at all levels here. Changes at the harness level might involve simple changes like prompts or tool schema changes, or changing the way that context is injected so that an agent given a certain task can answer in a certain way.

11:54 And then making changes at the model layer, I’ll kind of, I won’t dive into it too much because I know that Charlie here is going to cover some of the different post-training techniques that you can do at this level, but again, now you have the flexibility to be able to go in and choose your post-training algorithm of choice and make changes to help go and remediate whatever it is that you’re trying to test for and ideally mitigate. Once you’ve got that, then we can rerun and then we can establish that the environment and the agent running in that environment can pass those particular

12:27 tasks and begin to behave as we expect it to. Finally, once we’ve established a robust environment or world and set of tasks, it’s inevitable that as models develop, parts of what we test for will change and, you know, regressions will be introduced, but also parts of what we designed for to solve for particular deficiencies in our agent will be absorbed as part of the model. And I think that this is kind of no surprise to anyone, but we’ve

12:58 constantly seen parts of the harness, whether it’s, you know, long task decomposition or whether it’s dynamic tool routing, be absorbed into the model layer. And so, we’ll see that change. But we’ll be able to test for it because we now have a robust way to actually simulate how our agent is behaving in production. Also, what will be… also, as well, the interaction patterns of people using your agent will obviously change, and having a strong way to be able to go and test and actually roll out and determine whether or not your agent holds up under

13:29 those particular interaction patterns will still be available given that we have this really robust and powerful way to now test it. And so really being able to own your own intelligence comes down to being able to adapt to changes as models improve, as user interaction patterns change. And as builders, once we are aware of, and when once we have put the mechanisms in place to be able to go and test different behaviors through robust environments and through different tasks, we really then can stay on top of this fast world that we’re moving in.

14:00 And so, thank you for listening.