How to Build a Model Router in the Harness

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How to Build a Model Router in the Harness · Channel: LangChain · 13:16 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, Local and open-source models

Summary

Sydney from LangChain explains how model routing can lower agent costs by choosing a model suited to each task without measurably reducing quality. Using the Open SWE coding agent as a test bed, the team analyzed task traces, selected models across cost and intelligence trade-offs, and built a router that chose a model at the start of each thread. In an A/B test of almost 500 threads, routing reduced median cost per thread by 64%, with no statistically significant change in the measured quality outcomes. The video also discusses limitations and next steps, including offline evaluations, subagent model selection, and rerouting during a thread.

Key points

Model routing aims to match each task with an appropriately capable model, reducing cost without sacrificing output quality.
Understanding the agent’s task space is the foundation for designing routing criteria; the team used LangSmith traces for this analysis.
The team assessed task complexity using measures including median invocations and median LLM cost.
Models were selected along a cost-and-intelligence Pareto frontier and grouped into fast, balanced, and performance tiers.
The initial router ran once on the first message in a thread and selected the model for the rest of the task.
Task outcomes can be measured with offline evaluations or A/B tests, each with different trade-offs.
In an experiment with almost 500 threads, routing reduced median cost per thread by 64%; the difference in merged PR results was not statistically significant.
A follow-up test against a controlled fast model was stopped after users complained about being stuck on the fast tier.
Potential improvements include benchmarking on additional eval sets, choosing models for subagents, rerouting mid-thread, and gathering richer feedback.

Chapters

Start	Chapter	Summary
00:00	Why build a model router?	Sydney introduces the cost challenge of using frontier models and explains how routing can match models to tasks.
02:40	Understand tasks and available models	The team uses LangSmith traces to characterize Open SWE tasks and compares models by cost and intelligence.
05:18	Build the router and measure outcomes	Sydney describes the router criteria, its one-time model choice at the start of a thread, and the evaluation options.
08:34	Experiment results	The A/B test shows a 64% reduction in median cost per thread, while a separate fast-model test is stopped after user complaints.
10:38	Next steps and getting started	Sydney outlines possible routing improvements and recaps four steps for building and measuring a model router.

Quotes

reduce cost without sacrificing quality. — 00:00

we were able to see a 64% reduction in median cost per thread with no measurable change in quality. — 02:07

This is what you use as the foundation for building your criteria for the router. — 03:11

we didn’t want to significantly degrade user experience. — 10:07

make sure you have the power to measure success before you begin live routing. — 12:46

Concepts and entities

Sydney, LangChain, Open SWE, LangSmith, GLM-5.3-Flash, GPT-5.6 Sol, GPT-6 Astra, Jev, Artificial Analysis Intelligence Index, Deep SWE, Deep Agents, model routing middleware

Full transcript (original language)

00:00 Hey, folks. My name is Sydney. I’m the product manager for the open source team at LangChain. And today, I’m super excited to share with you how you can build a model router in the harness to reduce cost without sacrificing quality. Let’s jump in. So the motivating problem for building a model router is that frontier intelligence is really expensive. And so if you’re building agents that work at scale, this cost can be prohibitive. And so model routing can help to reduce cost, theoretically without sacrificing quality if it’s done well.

00:33 I have a quote here showing that model providers like Anthropic even recommend starting with smaller models, in this case like Claude Haiku, before graduating to a more intelligent model like Opus, because a lot of tasks can be done with these cheaper models. So I’m basically assuming here that we often don’t need the best model for a lot of tasks that we ask our agent to do. Past a certain point, you kind of hit diminishing returns where your cost and latency are increasing, but the quality of the outputs are not significantly better.

01:05 And ideally, an agent has good model-harness-task fit. And what does that mean? It means you have the right model for the job on the intelligence and cost curve here. And then your harness does a good job of getting the right context to the model for the given task at hand. So what does a model router actually do? It picks the right model for each task. And so a task request comes in, and a router has a classifier model, and then a prompt

01:35 with some criteria that help it decide what model would be most appropriate for a given task. So would that be a fast and cheap model, maybe something balanced, or a performance-focused model that’s going to be slower and more expensive? And then it selects the given model for the task, and then that model is used to power the agent that completes the task. And so our test bed here was Open SWE, our open source coding agent. And to kind of spoil the results of the experiment here, we were able to see a 64% reduction in median cost

02:07 per thread with no measurable change in quality. There’s four steps for building a model router, and I’ll explain them through the lens of this Open SWE experiment that we ran. But a little sneak peek here. First, you need to understand the tasks that your agent works on. You need to understand the models available for these tasks. Then with one and two done, you can build the router in your agent’s harness. And it’s very important to track task outcomes so that you can understand if your router is working well. That is reducing cost without sacrificing quality.

02:40 OK, so let’s talk about step one, understanding the tasks. The first thing that we did was try to understand what our coding agent Open SWE is being tasked with. This was all done using LangSmith. So every time our coding agent runs, it sends a trace to LangSmith that has a record of every step that that agent took. And so we were able to use all of those traces and do aggregate analysis over them to determine what types of tasks our agent was doing. So you can see here, unsurprisingly, for a coding

03:11 agent that code changes dominated things like feature requests, bug fixes, and tests. But there were also some things like design questions and general purpose questions. Out of all of the steps, I would actually say that this one is the most important. This is what you use as the foundation for building your criteria for the router. You know, deciding what types of tasks should be assigned which model. And so if you don’t have a clear understanding

03:42 of this task space, it’s going to be very hard to develop an effective router. And LangSmith, as I mentioned, was very instrumental in helping understand this space. I actually built a custom app on top of LangSmith to help me better understand this trace data. So you can see kind of the categories here and then active agent time and observed LLM cost filtered by different types of requests. Overall, this was super helpful for basically exploratory data analysis over my traces.

04:14 Also as part of step one, I wanted to better understand how complexity varied by task type, knowing that we probably want a more intelligent model for more complex tasks. So you can see here, we measured median invocations. So that’s the number of times Open SWE was called. And then also the median LLM cost for these different buckets of tasks. And then we generally rated a task as more complex if the number of invocations were higher or the LLM cost was higher.

04:46 And you can see kind of unsurprisingly here that like feature work was generally more complex than say testing work. All right, so this part is actually quite fun. Step two is understanding the models available for your tasks. So this data is pulled from the Artificial Analysis Intelligence Index as of September 9th, 2026, which is when we began this experiment. And you can see on the X-axis, we have the cost per intelligence task. And then on the Y-axis, we have the Artificial Analysis Intelligence Index.

05:18 This is what we call the Pareto frontier for LLMs. It’s this set of optimal models that have a good trade-off between like benchmark accuracy and evals compared to cost and latency. And so generally we want to select models along this frontier. And then given that we were designing a model router, we wanted to choose a small set of models here that had some trade-offs between their speed and intelligence index. The models that we ended up choosing were GLM-5.3-Flash for our fast model, GPT-5.6

05:53 Sol for our balanced model, and then GPT-6 Astra for our performance tier. Step three is building the router in the harness. So we used two sources of data to decide on the criteria for our initial version of the router. The first was the understanding of those tasks that we saw in step one. And the second was the official provider guides from OpenAI, for example, on which models to use for which tasks. And so you can see a couple of the descriptors that we used for each of the criteria for a given model.

06:27 Also important to note for our router here, the router just runs once on the thread’s first message and then selects the model for the remainder of the task. This is different from some routers which do this mid-flight routing. We decided to just do a simple version first, but when I talk about upcoming improvements or further work, mid-thread routing is covered. So when we initially built this router in early September, we used a small LLM, specifically GLM-5.3-Flash, to do the routing decision with structured output. But after the experiment,

07:02 Jev, a popular decision model, came out and had claims of being significantly faster and cheaper than LLMs with more constrained decision-making powers. And so we are now powering our Open SWE router with Jev, the decision model. And it’s much faster and much cheaper as Jev advertised. The last thing I’ll call out on this slide is that GLM-5.3-Flash is actually an open model. At LangChain a while ago, we wrote a blog about open models crossing a threshold.

07:33 We definitely think that’s true. We use open models a lot internally. And a model router is a great opportunity to experiment with using cheaper and faster open models for tasks that they’re more than equipped to handle. OK. And then for step four, you need to decide how you’re going to track your task outcomes. There’s two main ways you can do this. The first is through offline evals. So this is very safe. You have repeatable comparisons and experiments that you can run. But the catch is that it can be a little bit hard and expensive

08:03 to develop a representative data set. And so the faster approach here is to do an A/B test, where you know that your live traffic is representative. But the catch here is that we’re running this experiment for our live users, and so they may experience a non-optimal router, which is the risk. And so the two outcomes that we were tracking with our A/B test were the merged PR rate. And then also, we enabled users to provide feedback on Open SWE runs so they could either give a thumbs up

08:34 or a thumbs down. And so you can imagine we want to make sure that for the 50% of threads that run through this new router, the merged PR rate doesn’t go down significantly, and the feedback is not significantly different either. All right, so experiment number one, what were our results? We had almost 500 threads. And you can see that the median cost per thread for the router was much lower, 64% lower. The mean cost was also lower, and then the P90 cost per thread

09:05 was lower by 37%. We see actually for this case more threads from the routed version ending in a merged PR, although the difference was not statistically significant. And then we also see that the PR open rate was pretty comparable across our A and B branches. You can see on this breakdown kind of the proportion of tasks that were sent to each of the model tiers. So we had 34% sent to the fast tier, 56% sent to the balanced tier and only 10% sent to performance.

09:37 You can see on the far left, kind of the different distribution of costs for the always performance model versus the routed. And then you can see on the right side, for the routed threads, we had much cheaper fast runs and then the performance runs. And then on the right, you can see in aggregate, the fast runs were much cheaper and then balanced and performance more expensive accordingly. One interesting note, we definitely did see a couple of very long threads contributing

10:07 disproportionately to cost. And this was true for both the routed and non-routed arms of the experiment. I will note, we actually started a second experiment, which was router versus a controlled fast model. So in the first experiment, it was a control of the performance model. We did end this test within a day because we got so many complaints from users who were stuck on the fast arm. And so we got enough complaints. we didn’t want to significantly degrade user experience. And so we decided to stop this experiment. But it was pretty telling.

10:38 We certainly couldn’t switch over to GLM-5.3-Flash for all of the tasks that Open SWE needs to manage. So what’s next here? There is definitely a lot that we could do to further explore model routing optimizations. I think the first thing would be benchmarking the router on other eval sets like Deep SWE, rather than just doing the A/B testing, or even perhaps other benchmarks that are more review focused, because generally the task for Open SWE is to generate good, reviewable PRs. The second thing that we didn’t touch

11:09 is model selection for subagents. So there’s definitely more cost optimizations to be made if we also select appropriately sized models for the subagents that the main agent orchestrates with the Deep Agents harness. A third thing that I mentioned earlier we could certainly explore is rerouting mid-thread. So you can imagine if for Open SWE, a user asks a question and then discovers that maybe a pretty complex bug fix is in order or wants to design a new feature mid-thread, you might want to upgrade to a more capable model.

11:43 And rerouting mid-thread could help to ensure that the right model is chosen for the task at any point in the given thread. One trade-off here is that if you do switch a model, you experience a cache miss, which can be costly. But for async agents like Open SWE, there’s actually a lot of cache misses anyways, just because there’s more than a five minute idle time between when an agent responds and a developer has time to respond back in Slack. So this cache miss might actually not be a super significant concern.

12:15 And then the fourth thing would be digging for richer routing signals. So we didn’t get a ton of feedback on traces. We did get, you know, the PR merge rate was a very reliable metric, but trying to get more active feedback would be helpful as well. All right, so if you want to get started building your own router, I would recommend following the four steps that we did in this experiment. So one, understand your agent’s task space. Two, pick models along the Pareto frontier, the cost intelligence curve.

12:46 Third, treat routing as context engineering and build your router into the harness where the appropriate context is for the given tasks at hand. And fourth, make sure you have the power to measure success before you begin live routing. You can easily add routing to your own agent with our model routing middleware. And at LangChain, we’ve always prided ourselves on offering open source software that’s model provider agnostic. And so changing models in your router should just be a one line of code change.

13:16 You can also check out Open SWE at langchain-ai/open-swe on GitHub. You can learn more about building a harness with Jev, which is a decision model that’s incredibly powerful for the model routing step at the blog linked below. And we always love hearing from you. Feel free to tag us @LangChain on X. Thanks for watching and see you in the next one.