How to go from your agent's traces to a fine-tuned model in one workflow

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How to go from your agent’s traces to a fine-tuned model in one workflow · Channel: LangChain · 08:29 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models

Summary

LangChain introduces SmithTune, a CLI for turning LangSmith agent traces into supervised fine-tuning datasets and models in one workflow. The process preserves the context of agent actions, filters candidate trajectories using a rubric and model reviewers, and uploads approved examples to LangSmith. Users then prepare data splits, review training settings, fine-tune a selected model, and compare it with the base model on held-out examples. If the results meet the quality target, they can deploy the tuned model; otherwise, they can refine the data or settings and iterate.

Key points

Supervised fine-tuning teaches desired behavior through golden examples, including tool selection, workflows, error recovery, and when to ask for clarification.
LangSmith’s trajectory format captures what the agent saw at each turn, including the tools available to it.
SmithTune pulls candidate trajectories from a LangSmith tracing project and can apply filters.
A task-specific rubric guides three independent reviewers in deciding which trajectories to keep or drop.
Approved trajectories are uploaded to LangSmith as a dataset for supervised fine-tuning.
The prepare step selects a provider and base model, and creates training, validation, and test splits.
The plan command displays training settings for review before starting a paid job.
Training selects the checkpoint with the lowest validation loss and compares it with the base model on held-out test data.
Evaluation results are available in a LangSmith experiment comparison view, where users can inspect predictions and judge explanations.
The final workflow is to create, prepare, plan, train, evaluate, and deploy—or refine and repeat if results need work.

Chapters

Start	Chapter	Summary
00:00	SmithTune and supervised fine-tuning	The video introduces SmithTune and explains how supervised fine-tuning can teach agents desired behaviors using golden examples.
01:06	Selecting and reviewing agent trajectories	SmithTune pulls trajectories with their context, applies a task-specific rubric and model review, then uploads approved examples to LangSmith.
04:47	Preparing the dataset and planning training	The demo prepares a dataset with training, validation, and test splits, then reviews the model and settings before training.
06:53	Training, evaluating, and deploying	The video shows fine-tuning and comparing the model against its base model, then explains when to deploy or iterate.

Quotes

Supervised fine-tuning, or SFT, teaches a model desired behavior through golden examples. — 00:00

we need to have a strong representation of that. — 01:38

This helps us with our data separation and make sure that none of the examples that we train on leak into our validation or test sets. — 05:50

Create, prepare, plan, train, evaluate, and deploy. — 08:29

Concepts and entities

SmithTune, LangSmith, LangChain, Supervised fine-tuning, trajectory, Base10, Fireworks, Qwen3.8 27B, DeepSeek V4.1 Flash, GLM-5.3-Flash

Full transcript (original language)

00:00 Go from your agent’s traces to a fine-tune model in one workflow. Hi, it’s Jake from LangChain and today we’re launching LangSmith fine-tuning in public beta with SmithTune, a CLI to allow you to post-train models from your LangSmith traces in one workflow. Supervised fine-tuning, or SFT, teaches a model desired behavior through golden examples. For agents, that might mean choosing the right tools, following a particular workflow, recovering from errors, or knowing when to ask for clarification.

00:34 You might use SFT to specialise a large model for your application, distill a frontier model’s behaviour into a smaller model, or establish a starting policy for reinforcement learning. Depending on your goal and training data, it can improve task accuracy and tool reliability, or reduce cost and latency by teaching more efficient behavior. Getting there from your agent traces usually means manually stitching together several steps, selecting good traces, preserving their messages and tool context,

01:06 formatting training data, running fine-tuning, evaluating whether the resulting model actually improves, and then deploying that model if you like the results. In this video, we’ll walk through how Smith-Toon makes this process seamless in one end-to-end workflow. Let’s dive in. Firstly, it’s extremely important that we have a strong representation of the context behind each action that our agent took. In Langsmith, this is represented in the trajectory format, where we can see a detailed reconstruction of the agent’s trajectory as it went out to complete a particular task.

01:38 We want to capture what the agents saw at each turn, And so it’s important to have a representation where we can see everything that the agent referenced, including things like the tools available to it at that turn. This is particularly important because in some cases, for long-running agents, the tools available to the agent might change at each turn, and so we need to have a strong representation of that. Fortunately, Lang Smith’s trajectory format allows us to reconstruct this context in the exact way that we need for our post-training jobs. SmithTune starts with your Lang Smith tracing project, where we have sample trajectories.

02:11 From there, we’ll pull down those trajectories, optionally applying any filters to store them in a local directory. Next, you and your coding agent that’s using the Smith-Hoon CLI will work to define a task-specific rubric to help define what’s a good trajectory. This will help us keep the trajectories that we wanna use as our golden examples. That rubric will then be passed to a model council with three independent reviewers who will each decide whether to keep or to drop a trajectory based on the rubric that you’ve worked to define.

02:41 We then upload those traces back to LangSmith to have a created dataset that we’ll use in our supervised fine tuning jobs. Let’s type into using SmithToon. First, I’m gonna run the help command just to get a sense of what commands are available to me and to get a general layout of the CLI. I can see that I need to run SmithToonDoctor. So we can see that all my API keys and everything are set here. Another important part that I saw was that I need to confirm the data rights. I’ve gone ahead and done that already.

03:12 And so we can see that save version that I can reference for the remainder of using the CLI. Now I’m gonna start working with my coding agent that’s gonna help me in this dataset pull process. I’ve got this command copied here that asks Smithtoon to pull trajectories from my openSwee reviewer tracing project. This is an open source coding agent that we use a lot at Langchain. I want 50 target trajectories with a max candidate size for it to go and look through of 500. I’ve given it the project, workspace, IDs, as well as a filter so that it can help identify the correct traces. Great.

03:44 We can see that my coding agent is reasoning about the way that it needs to pull these trajectories. We can see that it’s downloading a candidate pool and then we’ll separately do a model council review after we define a rubric of what good looks like for us. Okay, we can see that our pull is complete and we’re ready for our council triage. I’m gonna ask it to help me define a rubric what good looks like for these trajectories. We can see that it’s asking me what the reviewer training set should prioritize. This first suggestion looks good. We can see that it’s helped me design a rubric.

04:14 This looks great. Now I’m gonna kick off the judging. I’m gonna use DeepSeq 4.1 Flash and GLM 5.3 Flash, so limit it to two judges. Both of them need to agree on keeping an example for us to continue to have it in the dataset. I’ll come back when this stage is finished. Awesome, we can see that we’ve got 50 trajectories that have been approved by both judges. I’m gonna now go and look at this report and just confirm that I’m happy with the structure reports being accepted. Yep, it looks great. So now I’m gonna tell Smith to my coding agent to please use the dataset push command

04:47 to upload this to Langsmith. I know that it contains some PII after looking through it. I’ll take care and remove that once it’s in Langsmith. Amazing, and we can see that the dataset upload has completed. I’ll go and check that out now. Great, we’re ready for the next step in our process. Now that we have our data set of golden trajectories, we’re gonna prepare it so that it’s ready for training. We’ll start from our created trajectories and then go and choose a provider, either Base 10 or Fireworks, as well as a base model. From there, we’ll create the training,

05:18 validation and test splits that we’ll use when we train our model and then when we evaluate it to see how it performs. Now we’ll run the prepare stage in SmithToon. We’re gonna specify our training provider, We’ll be using base 10. We’ll pass the workspace ID and the dataset ID for the dataset that we just created. And we’ll also pass the model that we want to use. In this case, we’re going to be using Quen 3.8 27B, which is a great model for supervised fine tuning. It balances speed and accuracy well and has a large sequence length so that we can train on longer trajectories.

05:50 I’ll check back in once this is finished. Awesome. Now that this prepare step is finished, we can see some high level information that’ll be useful for us before we kick off the next actions. That includes things like the model that we’re gonna be training, the sequence length who our provider is, how many examples were prepared, how many were rejected, and also understand a little bit more about the train and test split. In this case, we’re gonna train on 41 examples, test on five and keep four for validation. This helps us with our data separation and make sure that none of the examples that we train on leak into our validation or test sets.

06:21 Now for demonstration purposes, my data set size is quite small. If you were running an actual job, you’d wanna have a much broader range of test validation and training examples. Before starting training, we’ll run SmithTune plan. This shows the selected model, the number of examples, and settings such as learning rate, batch size, and epochs. Review these, then pass your chosen settings to the training command. Great, now that I’m nearly ready to train, I’m first gonna run SmithTune plan, just to make sure that everything looks in order before I kick off a paid training job.

06:53 This is gonna tell us some key information, as well as some other key configurations or the training job that will run. This all looks good to me and now I’m ready to kick off my training job. Now run SmithTune train to start fine tuning with base 10 or fireworks. In our case, we’ll be using base 10. During training, SmithTune checks performance on the validation set and selects the checkpoint with the lowest validation loss. Awesome. And so the plan that we generated earlier looks good and now I’m ready to start training. The CLI will select the checkpoint

07:24 with the lowest validation loss, compare it against our base model our held out test data and publish the results to Langsmith. The confirmed flag authorizes the page training and evaluation. And once our training job is finished, because we provided the evaluate command, we’ll be provided with a link to the Langsmith experiment comparison view to see the results of our fine tuning job. The base quen model is on the left, and the fine tune model is on the right. Each row represents a held out trajectory with individual replay predictions and judge explanations underneath.

07:57 The feedback chart summarizes agreement with the recorded reference actions and we can expand a row to inspect what each model predicted and why it passed or failed. Now this wasn’t a very large job. And again, I wouldn’t expect to see much of a benefit in the tune model that we have here. But we can see that our average agreement with the judge or golden trajectory is slightly higher on our fine tune quen compared with our base quen. And so promisingly, we can see the value that fine tuning this model is done for this particular task. Finally, we have the deploy step. If the results meet your quality target,

08:29 use SmithToonDeploy to serve the tuned model and connect it to your application. If the results need work, use the evaluation to refine your examples or training settings, then run another iteration. That’s the workflow. Create, prepare, plan, train, evaluate, and deploy. Try SmithToon on GitHub. The link is in the description and we’d love to hear what you build.