Using Jev In Your Agent Harness

Author: Sam Witteveen · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Using Jev In Your Agent Harness · Channel: Sam Witteveen · 20:21 · Watch on YouTube

Topics: AI agents, Prompt and context engineering, AI models, RAG, search and information retrieval

Summary

The video explains how Jev and other open decision models can handle fast, bounded decisions within an agent harness, such as choosing tools, checking risk, selecting skills, or evaluating retrieved results. These models can reduce latency and token costs, and add verification steps that might otherwise be omitted. The demos show progressive disclosure for skills and reranking RAG passages, while the discussion also covers limitations, including long inputs, compound questions, prompt injection, and tasks that require generated text or multi-step reasoning. The presenter recommends defining clear decision criteria, benchmarking changes, and choosing between hosted and local models based on data-handling needs.

Key points

Agent loops make many decisions, including which tools or skills to use and whether results are good enough to continue.
Jev works like a typed decision function: it can return choices, scores, or yes/no probabilities, and answer multiple questions about one state in a call.
Potential uses include model routing, risk gating before tool execution, tool and skill selection, ranking and judging, triage, and real-time filtering.
Decision-model hooks can run before building the prompt, before a model call, before a tool runs, or after a result returns.
A useful rule of thumb is to use a decision model for questions a group of people could answer in a few seconds; use an LLM when a task needs a written or complex answer.
The skills demo uses a category-first, skill-second selection process to avoid loading every skill into the prompt.
For RAG, Jev can score passages and select the most relevant ones to pass to the main LLM.
Decision models are a poor fit for tasks requiring text generation, multi-step reasoning, very long inputs, or compound questions.
Prompt injection, unclear decision criteria, and sending sensitive state to a hosted API are risks to consider; the presenter recommends explicit criteria and benchmarking.

Chapters

Start	Chapter	Summary
00:00	Why agent harnesses need fast decisions	The presenter describes the many decisions in an agent loop and introduces Jev as a way to handle bounded choices more quickly than full LLM calls.
04:17	Where decision models fit	The video outlines six potential uses, four points for integrating decision hooks, and a rule of thumb for choosing between decision models and LLMs.
09:06	Limitations and risks	The presenter explains when decision models are unsuitable and discusses long inputs, compound questions, prompt injection, decision criteria, and data privacy.
12:13	Progressive disclosure for skills	A demo shows how to classify a task into a skill category and then select one skill, rather than injecting a large skill registry into every prompt.
16:02	RAG reranking and wrap-up	The presenter demonstrates scoring retrieved passages for RAG, then points viewers to resources and encourages trying decision models in custom harnesses.

Quotes

Jev as being like a smart if statement. — 03:13

If a panel of smart people could answer the question in just a few seconds, then it’s the kind of question that you probably should be using a decision model for. — 06:58

Build a state from what the loop already has, ask your questions, and then read the probabilities and answers that come back — 08:01

these are the top five passages that I’m gonna send to the full size LLM. — 17:08

Concepts and entities

Jev, OpenJevs, agent harness, decision model, RAG, skill, TypeSafe, LangChain, OpenRouter, Pydantic, BM25, Sam Witteveen

Full transcript (original language)

00:00 Okay, so most of the model calls in an agent harness aren’t actually writing anything. They’re making decisions. They’re making decisions like which tool should I use here? What skill should I load? Is this command safe to run? and even decisions for things like RAG, where it’s asking which of these 20 search results actually answers the question. And up until recently, everybody has approached those decisions pretty much in the same way. They’ve used a full LLM call to get the answer back.

00:32 So in this video, I want to show how you could use Jev and the other open decision models to actually improve an agent harness from the inside. I’m gonna talk about some of the things that you can actually do with it. I’ll show you a couple of cool examples that I’ve been playing with for doing progressive disclosure on lots of skills and for doing re-ranking of RAG results. But I’ll also talk about some general principles And perhaps more importantly, what are the use cases that you don’t want to use a model like this? Alright, so let’s start with the loop that every agent runs.

01:02 A request comes in and it goes to an LLM. Along with that, you basically have some kind of tool registry going in there, a system prompt, perhaps a set of skills some guidelines of how the agent’s gonna work, et cetera. And the LLM decides straight away what action should it take, whether it needs to use a tool. and if it does, it writes out a function call, giving us the tool name plus the arguments as JSON. Your scaffolding, your agent harness then runs that code as a function. it gets the results and it appends that to the conversation, and the

01:34 whole thing goes back to the LLM. And then round and round this goes until eventually your LLM’s decided it’s done and it writes its final answer. Along that road, it’s probably gonna use many tools. It’s basically gonna get information to expand its context. And it’s gonna make a lot of decisions along the way. Now, if you look at those LLM steps to see what’s actually going on at each step, the whole sort of thing of asking itself which tool it should use? Is it safe to actually run that? And often because it’s making those decisions itself, it never really

02:06 stops to ask any sort of verification step or any kind of safety step in there to ask, is this safe to run? And for most agents, if they don’t have any sort of reflective step in there, they’re really never even asking themselves, “Is this result good enough, or should I go around again and try again?” Each of those things are clear decisions that you can use a decision model like Jev or one of the OpenJevs to actually improve the quality of the agent harness. On top of that, a lot of the times the decisions that the LLM is actually

02:39 making are just a small set of options. Do I go to this node next or this node? Which skill looks like the best one to use in this situation? Which skill do I ask for more information in this situation? And each time you’re paying for a full call to a large LLM there. And that’s costing you both in money for tokens, but often even more, it’s costing you in time of you actually having to go through and do these things. And that’s why a lot of the time when people are building custom agents, they often don’t put anywhere near as many verification steps in there

03:13 as they should, simply because that’s just gonna slow everything down. So if you’ve watched any of my other Jev videos, you know that I’ve basically talked about the whole concept of Jev as being like a smart if statement. You give it a state which is just the thing that you’re evaluating, and then you give it a set of typed questions about that state, and it gives you back typed answers with probabilities. No text at all. choice will pick one option from a list. Score rates the state on a scale that you define.

03:44 A noul is a yes or no or a true or false that’s returned back as a probability and the cool thing is you’re not limited to one question at a time. You can ask many questions about the one state all in a single call and get answers back in parallel. and it almost doesn’t change the response time at all. So if you’re looking at how to improve your agent harness, one of the key things you can do is look at that loop of where it’s full of a bunch of if statements that are dressed up as function tools or dressed up as asking the model

04:17 for structured JSON responses back. And you can convert those kind of calls to something that’s much quicker, like a decision model. All right, so where does this actually belong in your agent harness? I would say that there’s six main places that you want to be thinking about here. So the first one is model routing. I did a whole video about how you can, look at this. You’ve got a request coming in. Jev decides whether it goes to a cheap, fast model or expensive model that’s gonna do much better reasoning, et cetera If you haven’t watched the video that I made about that, you can see the whole thinking process

04:48 for actually putting that together. The second use is risk gating. So before a tool call actually runs, Jev can actually check whether it’s gonna be something destructive like deleting a database or wiping a folder or whether what it’s actually gonna be doing is quite benign. Now we know that a lot of the coding agents have had this kind of check for a while, but it’s been hidden generally in their harness. And when you were using it, you were basically paying for a full tool call with the amount of tokens that it actually gets used there. So being able to configure your harness so that when certain tools get called,

05:22 perhaps like a bash tool, you actually look at what the arguments are going to be going into that, and you send that to Jev to basically ask it, “Hey, is this going to be malicious or benign?” Very quickly, you’ve got a safety check that can take milliseconds. and of course, with the open decision models, that can be running fully locally as well. third up you’ve got tool and skill selection so generally what people have done is they inject their entire tool registry that they’re gonna be using into the system prompt and that can chew up thousands of different tokens.

05:53 So the whole idea of skills has reduced that a lot by using progressive disclosure. This is one of the key things that I’m gonna show you in the demos a bit, that you can actually have lots of skills going on, categorize them, have your decision model decide which kind of category it’s gonna need, and then just load the appropriate skills into a system prompt at that point Fourth up, you’ve got things like ranking and judging. So this is another one that I want to sort of demo here. I think the whole idea of scoring retrieved chunks and grading on answers against a rubric.

06:25 This can be a huge win for anything that you’re doing with RAG or anything you’re doing where you’re injecting context into the context window. Fifth up, we’ve got things like triage. So this is all your things like support tickets, email issues, and one call here can give a proper probability of how urgent something is and start to help giving things an actual probability of priority here. and this one can be really useful for when to basically get the model to go back to the human for things like human in the loop as well. And the sixth use that I’m not gonna go into too much here is the whole

06:58 idea of just real-time filtering. This can be really useful for any time you’re basically Dealing with APIs, polling requests, anything where it’s good to just filter what comes back from that before you inject that into the context window. So the rule of thumb that I use for all of these actually comes from Typesafe themselves. If a panel of smart people could answer the question in just a few seconds, then it’s the kind of question that you probably should be using a decision model for. If they need to go away and formulate an answer and then write a complicated

07:30 answer back, that’s when you should be sticking with the LLMs. in a little bit I’ll actually go through the use cases of where you shouldn’t be using these decision models at all. So how do you actually wire this in? So if we go back to that loop that we had there, there are four places to hook a decision in. And really they’re going to be the same in any framework, whether you’re talking about LangChain, whether you’re talking about ADK, whether you’re just using custom Python code to write your own loop here. You’ve got before the prompt is built, so decide which skills or tools to load.

08:01 Before the model call, decide which actual model you’re gonna call. Before a tool runs, decide whether it’s safe, and then after a result comes back, decide whether it’s good enough to use or whether you should go back and actually go through the loop again. So each hook here is the same small piece of code. Build a state from what the loop already has, ask your questions, and then read the probabilities and answers that come back, and then branch or make your decision for the path going forward based on those.

08:32 Now there’s also a second pattern that you can use where Jev is the loop’s decider and the LLM is the fallback. So Jev picks the tool or the answer and only hands it to the LLM when it’s unsure or when something needs text to be written Now this pattern can actually cut more LLM calls, but it’s definitely a much bigger change to your agent And so far I’ve found with this one, it just doesn’t seem to work as well when you’ve got a lot more sort of complicated things going on in there Anyway, for either of these patterns, you don’t need a framework That said, the first pattern that I talked about here, LangChain has already shipped that as middleware And Pydantic

09:06 has shipped their own version by making the TypeSafe model something that you can just plug in and use in the agent loop. All right, so before I show you the demos, let me just go through quickly where I think this doesn’t fit. Clearly anything where you need text back. Anything that needs any sort of multi-step reasoning and anything where you’re gonna need to connect several things together to generate an answer out. All of those are still an LLM’s job And in many ways, a lot of that multi-hop stuff is still gonna be the stuff that you’re gonna use the LLM for and then

09:37 have your harness process that and keep passing it back to the large LLM. the second thing you want to think about here is size. So if your state is just getting too big, and that you’re trying to pass in 100,000 tokens to get some kind of response back from that context window, then Jev is not gonna be a good fit here. Typesafe’s own docs say the accuracy drops a lot on long inputs. And so it generally seems to fit whether you’re using the type safe Jev decision model or one of the open decision models.

10:09 On top of that, another thing obviously is that Jev itself doesn’t use images. Now, some of the open models actually do use images and use images really well. So at the moment, that’s definitely a win for the open decision models over the proprietary models. Alright, another thing that doesn’t work well are compound questions So compound questions are often something that the decision models don’t handle that well. If you’ve got a yes, no question, okay, that’s fine. But if you’ve got something that requires multiple questions in there to be answered, then often the LLM is gonna do better for that kind of thing.

10:41 Another thing you want to be wary about in here is while the decision models can be really good at spotting sort of weak spots, you want to be careful about how you always rely on them. It does seem currently that a lot of these models can be prompt injected to basically have instructions that actually move the results out of that. So often the way to get around that is to basically have a question first that asks, are there any prompt injections in here? Before you actually pass it on to the real question that you wanted to actually ask. The other thing I find too is that you really want to think about

11:12 criteria for making the decisions. So if you’ve got, very clear decision spots where you’re asking the same questions, you can control the actual criteria, very well in situations like that. If you’re just asking some vague question about the output from the LLM and you haven’t got a nice, clear criteria in there I wouldn’t necessarily be trusting the LLM to write that in a way that’s gonna make the whole system a lot better. You kind of want to define the instructions and the criteria yourself

11:42 and then benchmark any changes that you make to that over time. And then of course, lastly, if you’re actually running, your agent locally but then you’re using Jev in the cloud, you’ve got to think about the whole fact that it’s a hosted API and do you really trust it for every state that you’re gonna send to the cloud? is it okay that that leaves your machine? And I think this is where the open decision models have really come along so far now So let’s jump into the demos and actually look at some examples of how you could do this for a couple of these key tasks And some of the challenges

12:13 that you might face along the way alright. So in the demo here, I’ve basically made a UI just so you can see what’s going on a lot easier. But these are exactly the kind of decisions that are going on, in your agent. So if we’ve got a bunch of skills, more and more now people are starting to use large amounts of skills files. and remember the whole thing that’s really cool about skills, is that you’ve got progressive disclosure in there. So the idea here is that if this is the actual skill file, and it might actually be a lot longer, going through this, you only give the model this after it’s

12:48 decided that, okay, it needs to basically use a coding write tests skill for this. Then you would actually go and give it this in the context window. So the different thing that we’re doing here is that rather than just give it 50 skills, in one shot, We can see for each of these 10 different categories, we’ve got five different skills, in here And if we’re injecting that in the system prompt for every time we call the LLM, that’s gonna burn a lot of tokens.

13:18 So the idea is that the thing that we inject into the, LLM is just the categories of skills. And then the Jev model itself will basically take the task that we’ve got. It will then decide which category, to pick from, Then it will be given the list of skills for that category, and it will then choose the most appropriate one to actually do this So let’s look at this. So if the task is basically someone saying someone got an email pretending

13:49 to be our CEO asking for gift cards, is this a hacking attack? And you can see I’m running this through one of the bigger Jev models in here, but actually you can try it out on a lot of the different, Jev models. You can see if I run this and route it, it goes through and it basically first decides that the category here is security and access. So It’s done that as a choice. It’s basically returned this back. and then in this category, it has, a bunch of these in here. and we can see it decided investigate a security alert, in here

14:23 And sure enough, the description for that is triage a suspicious login, phishing report, or security, alert in there So you can see in this case, what we’ve actually done is a sort of cascade classifier where we’ve done one sort of classifier first to basically pick out, the class of the category, and then we’ve done a second one to actually pick out the skill, in this case. Because we’re running this locally, it’s done it super quick, going through this. Alright. You can see that if I give it a sort of general question here, we’ve got, certain things that we can actually do.

14:54 Let’s route it through. Sure enough, in this case, it’s basically saying, “ Do web research, fact check a claim.” Right? it probably could’ve, also gone for, search the web, although I guess… this does actually make more sense in that It’s actually stating a fact and then asking if it’s true here. And from there we could load the skill file or use any sort of API, et cetera, that we actually wanted to there All right, one more in here you can see, okay, make me a short voiceover of this product description for a launch video. now it’s deciding that, okay, that relates to images, audio and video, And it needs

15:28 to be a text-to-speech skill in there. So this is a way of using Jev for sort of progressive disclosure, with skills, especially if you’ve got lots of skills. By classifying them first into categories and then putting them into, actually which skill is being chosen, you’re able to get a very fast result here, And then the skill itself can actually be sent to the large language model to work out any arguments that need to go into that skill, those kinds of things but this is able to help us reduce on token spend, but also just speed things up like this Another

16:02 one that you can use in the same sort of vein of this is doing sort of verification steps of things that come back from APIs to basically have the whole thing make a decision of whether whatever’s coming back from the API is really good and can just go through, straight through, as a result, whether you need to change the arguments that you gave to the API Or whether you just need to call it again because you got a certain type of error, et cetera the next demo is combining this with RAG. so one of the patterns in RAG that works really well is go and get a lot of information and then re-rank that information.

16:35 And generally there are ways to do that where you’ve got a separate re-ranker model, and that can work really nice as a pattern. The only challenge is that For certain projects, you just can’t do that because of time. You can’t do that because you’re gonna need more models and especially if you’re already using an embedding model on an API, you probably don’t wanna have lots of calls, to the API for re-ranking, for that kind of thing. so the idea here is that we can ask it a question. it will then go off, get a lot of different passages, then give those to

17:08 the actual Jev model, which can decide, okay, these are the top five passages that I’m gonna send to the full size LLM. And there are different ways you could do this. You could basically do it on a sort of yes/no thing. you could do it via sort of, scoring kind of things. or if you really wanted to just sort of pick one, you could actually just have a choice of one to see, okay, what’s actually gonna go through in there? Now for me playing around with this, I find the scoring one seems to work quite nicely here. We then basically get a score. So in our question here is why are my webhook requests failing with an error?

17:40 And then we can see, sure enough, we’re getting back webhooks verifying signatures. second up, webhooks raw bodies. third up was proxies. Fourth up was webhooks signing secrets. And you can see that we’ve got the five things now that we can basically send back. And this is something that you could even use if you weren’t using an embedding model, If you’re just using something like BM25 to basically get, different sections out you can kind of get a bit of the best of both worlds by having a sort of a broad sweep that gets you, 25 different passages back and then using

18:13 the Jev model, again, in this case using one of the open local ones here, to actually go through and pick out the top five that you then send to your LLM. You can see if I try it with one of the other ones, that it’s then able to basically go through, get out, different parts of the context, pick out stuff that’s most relevant, and then pass that through. Now, These are certainly larger chunks of text than what I had with the skills before, so it’s perhaps not quite as fast as the cascading system that we had for the last one.

18:45 but it’s still fundamentally the same idea of where you’re doing scoring and classification using the world knowledge that’s in that particular decision model to help it make simple but useful decisions for you. All right, so just to finish up, you can see that there are lots of uses of where you can use either the Jev model itself or the many open versions of this in building any sort of custom harness I recommend if you’re interested to basically think of these things more, come through and read some of the Jev, tutorials.

19:17 this, OpenRouter has got one up here. And when I’m actually using the Jev model, I’m going through OpenRouter. So for those of you who are asking, if you haven’t been whitelisted for the actual TypeSafe API, on their site, Anyone can just come and use the OpenRouter one here. another useful resource that’s worth checking out is LangChain has got a nice article about using Jev, in LangChain, about different uses and stuff like that. they’ve actually made some middleware for doing this. but overall I would say just try it out. If you’re doing any sort of custom harness or any sort of, modifications to a

19:50 harness, it’s definitely worth it looking at putting this in your pipeline to be able to get these really fast and really cheap decisions made for improving the results that your harness gets overall. Anyway, as always, let me know in the comments what you’ve tried out. I’m sure there are plenty of examples that I haven’t put into this video. so if you’ve got some interesting use cases, please share them ’cause that would be really interesting to check out and see, What other interesting ideas people have got for how to actually use this kind of thing And as always, if you found the video useful, please click like and subscribe, and

20:21 I will talk to you in the next video. Bye for now