How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How Lyft Increased Its Agent Resolution Rate by 16% with LangSmith and LangGraph · Channel: LangChain · 03:12 · Watch on YouTube

Topics: AI agents, Task and workflow automation, AI evaluation and benchmarks, Software engineering with AI

Summary

Lyft describes handling a broad range of support issues for riders, drivers, and applicants, with resolution rate as a key metric. Before adopting LangSmith, its agents were basic and difficult to evaluate, observe, and debug in production. Lyft created a self-serve agent-building platform where PMs and operations teams can configure agents using prompts in LangSmith Prompt Hub, while engineers review them. With LangGraph’s meta-agent approach and LangSmith tracing and evaluation tools, agent development time fell from six months to one or two weeks, and the resolution rate increased by 16%.

Key points

Lyft’s customer support covers use cases such as lost and found, damage claims, and taxes.
Its support serves riders, drivers, and applicants, and resolution rate measures how many issues can be solved without escalation to human agents.
Before LangSmith, agents were largely deterministic, with a single-shot LLM, a large prompt, and access to internal APIs.
The team lacked proper evaluations and had difficulty identifying whether failures came from tools, prompts, or hallucinations.
Lyft built a self-serve platform so PMs and operations staff could create and ship agents with limited code changes.
Agents are registered as nodes in a meta agent and loaded dynamically when the service starts.
Agent configuration is stored in a runtime repository, while prompts are kept in LangSmith Prompt Hub and reviewed by engineers.
LangSmith tracing, debugging, evaluations, and testing made the development process smoother.
The time to ship agents dropped from six months to one or two weeks, and the resolution rate increased by 16%.

Chapters

Start	Chapter	Summary
00:00	Lyft’s support scale and resolution goals	Lyft outlines its broad support needs and its goal of resolving customer issues without escalating them to human agents.
00:33	Challenges with early agents	The team describes basic agents, limited evaluation and observability, and difficulty debugging production behavior.
01:37	Building a self-serve agent platform	Lyft explains how non-engineering teams can configure agents with prompts that engineers review before deployment.
02:07	Dynamic agents and faster shipping	A meta agent loads configured sub-agents, while LangSmith tools help reduce agent development time and support a 16% resolution-rate increase.

Quotes

One of the metrics we go for is what’s like the resolution rate. — 00:33

So it was it was difficult to debug issues. — 01:05

So, it’s merely like a config change — 01:37

that has reduced to from 6 months to a to a few weeks, uh 1 to 2 weeks. — 02:41

increase in resolution rate by 16%. — 03:12

Concepts and entities

Lyft, LangSmith, LangGraph, LangChain, LangSmith Prompt Hub, meta agent, resolution rate, observability

Full transcript (original language)

00:00 Customer support use cases is very broad. It can be like issues like lost and found or issues like damage claims or issues like taxes. We have like 500 million type of interactions every day and we have almost like 200 to 300,000 user queries per day. Which is a lot. So we work for riders, drivers and applicants. One of the metrics we go for is what’s like the resolution rate. So how much how much can we solve our customer issues without

00:33 escalating them to human agents. Our platform covers about 30 to 40% of use cases before we adopted LangSmith, our agents were pretty basic. They were very deterministic. Maybe like a one single shot LLM file with a huge prompt and a bunch of internal APIs it can access. >> [music] >> So there was no like sort of sophistication in multi-turn conversations. There were no evaluations, proper evaluations. The observability was a was a big issue because we we kind of launched a few agents in

01:05 production and sometimes it did not behave as we expected it to be. But then we didn’t know what to do about it. Like we didn’t know which component inside that pipeline broke the response, whether it was a tool call or like a prompt or maybe like LLM itself hallucinated in between providing a response. So it was it was difficult to debug issues. It was getting difficult to scale. So we wanted our engineers to focus more on

01:37 more complex use cases or maybe like foundational improvements and not spend all of their time building agents. We tried to bridge this gap by creating a self-serve platform for our non-engineering people like PMs and ops where they can use their own domain and natural language prompting skills to build agents and ship them to production without making a lot of code changes. So, it’s merely like a config change and

02:07 they they write some prompts on LangSmith prompt hub. We as an engineers can review them, but they are like very easily deployable to production. [music] So, what we did was we have like a meta agent and all the sub-agents are kind of registered as nodes in that meta agent. And we we do that dynamically when we start the service. So, every new agent now it’s just like a config which we have in a runtime repository and prompts which we have in the LangSmith prompt hub. So, we just pull that and like the the

02:41 platform composes rest of the things. So, it’s become pretty easy and because of LangSmith tracing, the debugging and evaluations, testing, everything has become like very smooth and very easy. It was a huge upgrade to ship to production. But then with this agent builder platform [music] that we have created, that has reduced to from 6 months to a to a few weeks, uh 1 [music] to 2 weeks. And that’s pretty exciting because now like we have free time to focus on more complex and foundational improvements

03:12 while PMs and other ops people can directly ship agents. So, it’s it’s the iteration loop has become faster. Shipping faster has enabled us to create more agents and like cover more issues in production and that’s definitely resulted in >> [music] >> uh increase in resolution rate by 16%. So, yeah, since we have started using LangSmith [music] and LangChain.