Vanta and LangSmith: Assessing Compliance Evidence Like an Auditor
Read the Portuguese translation →
Author: LangChain · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Vanta and LangSmith: Assessing Compliance Evidence Like an Auditor · Channel: LangChain · 01:01 · Watch on YouTube
Topics: AI evaluation and benchmarks
Summary
Section titled “Summary”The speaker describes a capability for assessing compliance evidence in the way an auditor would. They explain that finding traces with other vendors and in-house tools was frustrating, while LangSmith’s observability features make it easier to search traces and metadata. They also praise LangSmith evaluations, especially align evaluators, for encouraging clear judgments about ground truth. The speaker says LangSmith has made their workflow much faster and supports improving evaluator and AI quality outcomes end to end.
Key points
Section titled “Key points”- The team is building a capability to assess compliance evidence like an auditor.
- The speaker struggled to find traces using other vendors and in-house tooling.
- LangSmith observability supports flexible trace browsing and full-text search.
- Users can search through metadata fields to find relevant data.
- The speaker is a fan of LangSmith evaluations, especially the align evaluators feature.
- The evaluation workflow asks users to label ground truth as good or bad.
- The speaker says LangSmith has made their workflow 10 times faster.
- LangSmith is described as focused on improving evaluator and AI quality outcomes end to end.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | Assessing compliance evidence | The speaker introduces the compliance-evidence capability and describes the difficulty of finding traces with other tools. |
| 00:30 | Observability and evaluations | The speaker explains how trace search, metadata, and align evaluators help find data and label ground truth. |
| 01:01 | Faster quality workflows | The speaker credits LangSmith with speeding up their workflow and supporting end-to-end quality improvement. |
Quotes
Section titled “Quotes”We’re building a capability that helps assess compliance evidence the way an auditor would. — 00:00
I would just spend 10 minutes just trying to find something and giving up. — 00:00
I can finally find the data that I want without having to fight my tooling. — 00:30
it forces you to have an opinion and label your ground truth like — 00:30
made my workflow 10 times faster. — 01:01
Concepts and entities
Section titled “Concepts and entities”Vanta, LangSmith, LangChain, observability, traces, align evaluators, compliance evidence
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 We’re building a capability that helps assess compliance evidence the way an auditor would. One of my pain points with other vendors and our own in-house tooling is just inability to find things that I wanted to find. I would just be looking for some trace. I would just spend 10 minutes just trying to find something and giving up. LangSmith really nailed the observability features. LangSmith observability allows you to really
00:30 flexibly look through your traces, do full text search, and look through the metadata fields. I can finally find the data that I want without having to fight my tooling. I’m a big fan of LangSmith evaluations. Specifically, I really like the align evaluators feature. I think what the product gets right is it forces you to have an opinion and label your ground truth like >> [music] >> good or bad. Is this good or is this a loss case? And there are just certain affordances in the UI that I feel like
01:01 has made my workflow 10 times faster. I feel like LangSmith gets it. They really get that we want to improve these out like evaluator and AI quality outcomes end to end, and they’re thinking rigorously about what tooling number five.