The 3 Things Clay Measures on Every AI Agent

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: The 3 Things Clay Measures on Every AI Agent · Channel: LangChain · 00:32 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization

Summary

Clay evaluates its AI agents using three measures: run quality, throughput, and cost. LangSmith is used for offline evaluations and benchmarking quality against cost and throughput. Quality checks include comparing answers with ground truth, checking deterministic matches in structured outputs, and using LLMs as judges. Clay customers also use agents to combine research with synthesis for recommendations, including policy and alpha messaging.

Key points

The three measures Clay considers are agent-run quality, throughput, and cost.
Quality includes whether the agent returns correct data.
Throughput concerns how many runs can be completed efficiently.
LangSmith is used for offline evaluations and benchmarking.
Some evaluations compare answers against ground truth.
Other evaluations check deterministic matches in structured outputs.
Clay also uses LLMs as judges for some evaluations.
Customers use agents to combine research and synthesize it into recommendations.

Chapters

Start	Chapter	Summary
00:00	Three measures for agent runs	Clay considers quality, throughput, and cost, and uses LangSmith to evaluate and benchmark them.
00:32	Ways to measure quality	Evaluation methods include ground-truth comparisons, deterministic matches, structured outputs, and LLM judges.
00:32	Research and recommendations	Clay customers use agents to combine research with synthesis for recommendations.

Quotes

we think about the quality of the agent runs. — 00:00

We think about throughput. — 00:00

And we care about cost. — 00:00

Some of these are just we look for deterministic matches — 00:32

we do use LLMs as a judge. — 00:32

Concepts and entities

Clay, LangSmith, quality, throughput, cost, LLM as a judge, Agent

Full transcript (original language)

00:00 We think about three three things to calibrate. So, we think about the quality of the agent runs. So, actually people care a lot about getting correct data. We think about throughput. So, how many of these can we run in an efficient way? And we care about cost. And so, I think we’ve used tools like LangSmith to really uh do offline evaluations of where we are in quality and benchmark those against kind of where we are in a cost throughput position. >> How do you measure quality for these runs? >> We have a mix of different methods. Some of the evals that we have are just purely factual, right? So, just is it is the agent actually able to get the right answer and compare those against ground

00:32 truth. Some of these are just we look for deterministic matches and we kind of structure outputs and and sort of uh do it that way. Um and for some of these we we do use LLMs as a judge. Um I’d say a lot of the Clay our Clay customers um will also use these agents for combining both research and then actually synthesizing that research into a recommendation, whether that’s a policy nation or alpha messaging. It’s a multi-purpose sort of uh agent.