How To Build A Harness With Jev | A LangChain x TypeSafe Conversation

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: How To Build A Harness With Jev | A LangChain x TypeSafe Conversation · Channel: LangChain · 47:40 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Prompt and context engineering, Software engineering with AI, AI models

Summary

The conversation introduces Jev as a type-safe System 1 decision model for fast, well-scoped questions, rather than a text-generating LLM or a standalone end-user product. The speakers explain its choice, score, and Noul question types, and discuss why machine-to-machine software can benefit from fast decisions without routing every task through a large language model. Hunter demonstrates integrating Jev into LangChain agent middleware for risky-tool checks and model routing, with tracing in LangSmith. The discussion also covers practical guidance on auditing and refining Jev instructions, use cases such as real-time filtering, and limitations around confidence interpretation and large contexts.

Key points

Jev is an API product that makes fast decisions from well-scoped questions using type-safe schemas; it does not generate text.
System 1 models handle quick, intuitive determinations, while System 2 involves multi-step reasoning; code can compose many decisions into a larger process.
Jev supports three question types: choice selects an option, score rates a state along a defined axis, and Noul answers whether a statement is true.
Questions should be narrow and non-compound, and score levels should have clear semantic definitions.
In agent harnesses, Jev can classify potentially risky tool calls before execution and route tasks to different models.
Tracing and auditability help developers inspect Jev calls; keeping instructions and criteria centralized makes them easier to refine.
Jev’s speed and cost can enable real-time applications, including filtering high-volume chat streams.
Confidence is not a universal signal for whether to take an action; developers should test their use case and choose suitable probability-based logic.
Large or unnecessary state inputs can reduce accuracy, so developers should include only relevant information and test their questions.

Chapters

Start	Chapter	Summary
00:02	LangSmith introduction	The opening presents LangSmith and its agent development lifecycle of building, testing, deploying, and monitoring agents.
01:44	Introductions and what Jev is	The speakers introduce themselves and describe Jev as a type of model for fast, scoped decisions rather than text generation.
05:22	System 1 models and machine-to-machine software	Allie contrasts fast System 1 decisions with System 2 reasoning and explains Jev’s role in software systems that communicate machine to machine.
12:26	Jev’s question types	Allie explains choice, score, and Noul questions, emphasizing clear criteria and avoiding compound questions.
17:29	Adding Jev to an agent harness	The LangChain team explains why Jev is not a direct LLM replacement but can address specific decisions within an agent harness.
20:05	Tool safety and model routing demo	Hunter demonstrates using middleware to block a risky customer deletion and route different tasks to appropriate models.
26:22	Observability and development tips	The speakers discuss tracing Jev calls and keeping prompts and decision criteria accessible for inspection and improvement.
30:36	Context limits, roadmap, and use cases	Allie discusses Jev’s context limits and unspecified future improvements, while highlighting real-time applications such as chat filtering.
37:50	Training data and confidence	The discussion covers Jev’s synthetic training data and why confidence thresholds and probability outputs need to be evaluated for each use case.
43:31	Long contexts and closing remarks	Allie advises limiting state to relevant information, and the speakers close by encouraging developers to test and share their Jev projects.

Quotes

Jev is about making fast decisions from specifically well-scoped questions with type safe schemas for each of them. — 03:20

system one thinking is the like, it’s the fast intuition side of thinking. — 05:22

Noul is kind of the simplest. — 16:03

tell your agent to put all of the copy, all of the instructions, criteria, anything that’s being fed into those into one centralized place — 27:57

make sure you’re only providing in the state what is actually necessary for making the determination. — 44:33

Concepts and entities

Jev, System 1 models, System 2 thinking, agent harness, LangChain, TypeSafe, LangSmith, choice, score, Noul, Jevify, Model Jaggedness, Allie, Hunter, Sydney

Full transcript (original language)

00:02 Agents are new kind of software. Endless inputs and not deterministic outputs. Build them the old way and they break. Enter LangSmith, the agent engineering platform. It’s model agnostic, cloud agnostic, and framework agnostic. Organizations shipping the best agents iterate with a system, and we call that system the agent development life cycle. Build, Test, Deploy, and Monitor. You build with our open source frameworks.

00:35 Deep Agents, LangChain, or LangGraph. LangSmith Fleet enables anyone to build agents. Without code. Test agents using evals and experiments. Deploy changes in one click with LangSmith deployment. Monitor every interaction from a single dashboard.

01:05 Governance is built into every stage of the agent development cycle, with LangSmith’s LLM Gateway. LangSmith Engine helps you improve agents autonomously and move through the development lifecycle more quickly. LangSmith Engine is your agent for agent engineering.

01:44 - Folks, welcome. We are super excited to get started with this building a harness with Jev, LangChain and TypeSafe webinar. Yeah, so first I wanna introduce our two guests today. Allie from TypeSafe and Hunter from LangChain. Do you guys wanna introduce yourself? - Sure, hi, I am Allie. I am developer relations at TypeSafe. You see my name all around our Discord.

02:15 If you’ve been in there and you’ll be seeing a lot more me and more demo videos and generally just out here teaching people how to use TypeSafe, how to think in TypeSafe, how to think about System 1 models in general. And yeah, super excited to be here. Yeah, and then I’m Hunter. I’m one of the technical leads on the open source team here at LangChain. Super excited about Jev. I’ve been pretty much nonstop experimenting with it over the weekends. Yeah, super exciting stuff. and I’m super glad to be here and talk to everybody.

02:49 - And I’m Sydney, the product manager for our open source team at LangChain, close partner with Hunter on all of our recent Jev work. Yeah, I think everybody’s lost a little bit of sleep in the past week experimenting and yeah, learning all there is to learn about Jev, so we’re super excited to chat more about it today. Allie, maybe you could start, Could you kind of explain in simple terms, what is Jev? Why is everyone so excited about it?

03:20 - That’s gonna be the hardest question to answer this entire time, is like how to explain Jev in simple terms. So Jev is an entirely different class of model from when we’re talking about, most of the time today when we’re talking about models, we’re talking about LLMs. And Jev is different from that. Fundamentally, Jev does not generate text. It is not a generative AI in that sense, where it is to create text or create images.

03:50 Jev is about making fast decisions from specifically well-scoped questions with type safe schemas for each of them. So Jev fits into the overall picture of AI, where we’ve seen over the past couple of years a huge increase in using these very heavy LLMs to do what is ultimately like a tiny decision, like, oh, which department should this support ticket

04:20 be routed to? Or is this email a promotion, or is it spam, or is it whatever? Or is this tool call like a safe tool call, like does it perform any right operations, stuff like that? And so, Jev is a new class of model that we call System 1 models, and it fits into that picture. Then the thing I always like to say, although this is a LangChain audience, so probably don’t need to say, but it does confuse some people. Jev is not an end-user product out of the gate. It is an API product.

04:51 It does not do anything until a programmer, whether a meatspace programmer or an AI programmer, until it puts that into software. That’s the first question that comes up is, how do I just point OpenCode at Jev? And it’s like, well, it’s a little more complicated than that. Like you use it inside your software to make what I like to call smart software. Yeah, I think that was a great simple explanation of Jev. You mentioned it’s called a system one model. What does, what does that mean?

05:22 So it comes from, and I always forget the author’s name, but there’s a book called Thinking, Fast and Slow. And, uh, and the author of that came up with this way of labeling like kind of two different ways of thinking. And system one thinking is the like, it’s the fast intuition side of thinking. It is quick determinations, where system two is like multi-step reasoning thinking, right? You have to think about this and put it together with that and connect the dots and then you have output. And so system one model, like Jev,

05:55 is a fast intuition based model. It is about, I like to describe it sometimes to be like, The questions you should ask, Jev, you should be able to present it to a panel of super smart humans and they could answer in five seconds, right? It’s not a you have to dig through and make multi-step reasoning arguments to understand the output. It is very, very smart, but the system one thinking type of questions is a different type.

06:26 It is a subset of all of the problems you might have in the world. Yeah, yeah, that’s a good explanation. Would you say that like traditional LLMs kind of fall into that like system two thinking? It’s a good question. I think before reasoning models, I think LLMs were much more like system one thinking in general, right? They would just kind of like answer quickly. But it was very intuitive and oftentimes like not trustworthy answer back in those days.

07:01 Now with reasoning, I would call that like very much, oftentimes that is system two thinking, right? When you send off a request to like a big frontier LLM, right, it’s going and it’s like looking, especially if it’s in an agentic loop, it’s like looking things up, it is making connections in order to like, to find that. But what I also like to think of as like, the system two oftentimes, that is integrated with our system one is your code. Like code, a lot of code is the system two thinking, It is the putting pieces together, connecting them together,

07:35 composing a bunch of system one decisions into a sort of reasoned final output. Yeah, that makes sense. You mentioned a couple of times, like, Jev is really helpful when you like embed it in code. Can you say a little bit more about like, what kinds of decisions Jev is good at making in code and kind of why it’s this like code first model. - Yeah, so I’ll start with the end of that

08:05 where you talked about it, like why is it a code first model? And a lot of that just comes from us seeing what is needed in the world, right? We see the future, and I think this is probably not even, like, I bet it’s not controversial to say that like, the vast majority of automation in the future will be machine to machine automation, right? is machine to machine communication. Already a huge amount of the world’s infrastructure is machine to machine communication. Like API is talking all over the place and a whole bunch of things working together, right?

08:37 So that’s how like most of our digital world works is code talking to other code. And so, Jev, like the system one model concept and Jev is about fitting into that picture. It is like we already need our computer to talk to another computer. So when we wanna bring the incredible power of AI, the intelligence that we want to access from these models, why shouldn’t it be a machine native format to begin with?

09:10 And that is the foundational thinking behind why Jev is shaped the way it is. And to expand on it a little bit, I’ve for a while, I like developing with LLMs, I like putting LLMs into these decision points. Before I started working at TypeSafe and saw this, I was using it for a lot of those. But it always, there was always an itch in the back of my head that I thought, isn’t this absurd? Like, isn’t this completely zany that we are,

09:40 we have this big, huge, heavy machine, this LLM machine that is doing a huge amount of computation to output human language, to output text. And then we use that to do tool calls or invoke CLIs, for example, which themselves, a CLI is a human interface. It was designed to allow a human to interact with a computer system in a way that’s semantic and that we can memorize.

10:12 But ultimately it turns it back into just raw machine code and goes off and does its thing. So we’ve got this weird bridge in the middle where to get a computer to use a computer [audio drops out] [audio drops out]

10:52 down into sanity, coming back into reality of if it’s machine to machine, there should be no prose-generating model in the middle of that, unless you need prose generated for your output. That’s fine. That’s still a huge place for LLMs to sit in all of this, but there’s a lot of places where it ought not to be. Yeah. I feel like, I think Hunter shared this analogy recently. It kind of feels like we, you know, LLMs became popularized and we were like,

11:23 “Oh my gosh, like this is a hammer tool. Like everything looks like a nail, right? Like let’s just use this for everything. It can do a good enough job.” But often they don’t like fit super well in code. And so we kind of made these like tack on bridges, like tool calling and structured outputs to be able to support LLMs in code driven applications. Specifically the structured outputs, I think they were pretty transformative in the early LLM days. Like, okay, it produced text and then also something

11:54 adhering to a JSON schema. But again, way over investing in the computation. And that shows both in the like cost and latency that we see from LLM responses for a lot of, I think pretty simple decisions. So one other question that I had, Jev has kind of three types of like decisions that it can make or questions. Maybe you could talk through those. I think I have a slide here. Yeah, could you talk through kind of these?

12:26 - Yeah, so this is like, this is the actual manifestation of like that machine to machine communication, right? The type safe question. So when you do a call to Jev, you are giving it like a state as input. That’s what you got up there. The high I’ve been trying to connect my Stripe account for three days, right? So the state is like the thing that you are evaluating. And then you give it an array of questions. You can give it one, you can give it 100 of them, but each of the questions is one of three types.

12:57 So we’ve got choice here, which is to pick one option from a list. So in this example, it’s like, oh, which team should handle this? Billing, technical or sales, right? So it’s like, it works best when there’s like going to be one dominant answer and the choices don’t have a huge amount of overlap, But when I say it works best, there’s always like a gazillion different like nuances and ways to talk about this. And we like, even our docs only scratched the surface and we’re like adding to it as we learn from the community. But so a choice is to pick one option from a list.

13:28 It’s like, it’s the closest analog to like what people would call a classifier, right? It’s just like, which of these is this thing? And I would say choice is like probably the question type I use the most, like maybe second to Noul. Score is about rating the state along an axis. So you’re gonna get back a value from like zero to N. You define how many levels there are. Let’s just pretend we were doing zero to four, okay?

13:59 So, or actually here we go. We got zero to two, calm, frustrated, very angry in this example here. So you’re saying how frustrated the customer appears. So score is all about rating the state on a single axis of determination. And that’s like the most important thing to know about it when using it. And it actually goes for kind of all of our questions is you really need to break down the questions to the point where they can’t be broken down anymore. Make sure you’re not asking a compound question.

14:29 Don’t put multiple different like axes of criteria in a single score. You should do two scores if there are actually like two axes of information. So the huge power from score comes from the way that you define the levels. So we don’t see it here in this like quick overview, but the calm, frustrated and very angry, each of those is defined by semantic description of what those levels are. In this case, it might just say calm, frustrated, very angry,

14:59 but it’ll always work best if you like give it, you should be able to like clearly define what a one means in your scale. And again, it’s like giving it to a panel of humans, right? It should be defined well that a panel of humans could be like, oh, one is very well-defined, two is very well-defined. And what that means is when you use that same question with the same criteria to evaluate a whole bunch of different states, maybe they’re a bunch of documents, maybe they’re live chat messages, there’s a whole bunch of different things. It could be log lines, output in an observability system.

15:33 you can trust that if you got a score of like one or 1.5 or two across multiple different things that they mean the same thing, that like because they meet the same well-defined criteria, it makes it so much more powerful than just like a blind scale from like, if you just did like level zero is like, you know, calm and level 10 is very angry with no descriptions in the middle, you’d be like, well, what it actually is a six, what is a seven, what is a four? Like, can you really be sure that that has a firm description?

16:03 So that’s score. And then the final one is Noul. Noul is kind of the simplest. It really is just asking if the answer to a question is yes, or if the statement is true. So you just ask like a simple straightforward thing, like the message conveys urgency or time sensitivity. There’s always like a little bit of a yellow flag if you’re putting an or in a Noul, ’cause that’s like, oh, is that actually asking

16:58 like a Boolean answer, but it’s not a binary state. It is continuous from zero to one. - Nice. Yeah, I just saw the Bernoulli thing before this, but I like your take better. So at LangChain, we think a lot about, how can we help people build agents? And one of kind of our biggest foundational pillars at LangChain is like, we wanna help people build agents with any model, right? the model that’s right for one task

17:29 is not gonna be right for another. And then also, you know, there are new models every week and these new models are good at different things. So we were very excited to see Jev come out and help support people building agents with Jev. Maybe Hunter, you could talk a little bit more. This is a bit of a two-prong question, but how do we see Jev fitting into agents and kind of how can you build it into your agent furnace? I can let you screen share as well as if you want. - Yeah, I got a little bit of a demo cooked up,

17:59 which I think I can use to demonstrate better in words. But yeah, at LangChain, we’ve been talking a lot about what makes a good agent harness, how do we get the most useful output out of a model? Usually the way that we interface with that, just because that’s how other people interface with models is some kind of opinionated to harness the agent loop, that sort of thing. We develop a ton of opinions about how we want that best to look.

18:30 When Jev first came out, we initially thought of it as there’s a couple of super interesting problems that we can solve directly inside of the harness. Just straight to do, there’s questions that we wanted answered, and not that great of a way to do it because if we threw LLMs to make decisions for us at every step, then gets very expensive, very lengthy. There’s just like a ton of complications that come with it. So let me and pray for me.

19:02 I know that live demos are a curse, but we’re going to try it anyways. Let’s see. So– Just– Oh, go ahead. Yeah, do you– something I want to say I can– I was going to start another demo, but– Yeah, I was just going to say to add like a little bit of color, we think about the core kind of agent loop as like send a request to a model, an LLM to be specific. And then the LLM can kind of reason, call tools

19:33 and do this like tool calling loop until it decides it’s done with a given task. And as Allie mentioned, like you can’t just plug in Jev for like an LLM as a direct substitute. That’s not the way that Jev is designed both in terms of its like functionality and API. But yeah, Hunter is gonna kind of cover how we think about plugging it into the harness because it is still very useful in agentic applications. - Yeah, for the agent harness specifically, there’s a couple of points. If you are anyone that’s used like codex or Claude code

20:05 or something of that variety, you might notice that when it takes certain actions, it usually classifies it as risky. I think everybody’s heard the like pattern case where an agent goes and deletes a production database and then all of a sudden everything’s on fire and we have to blame the LLM for it. Sort of what Claude and Codex adopted pretty early was to have some kind of classification step. This was like abstracted away behind their harness and how they implemented it to classify if a decision was risky or not.

20:37 Now with Jev just because it’s so cheap and inexpensive, we can leverage a very similar paradigm just in the agent harness. So this like basic example I have here, and this is very LangChain Forward code, if you guys are familiar. You can go look at our docs. I think there’s a lot of concepts here. We can chat about separately too, but yeah, just wanna, so like to go front to back, we have these two tools right here. You can see that we just are updating some thing in CRM,

21:10 or we’re deleting a customer account. We can classify this as like a pretty destructive action. We usually want like some explicit intent before this happens. We don’t want an agent to just do this automatically. The way that we can add this into the agent, LangChain agents have this notion of middleware, which is basically just at different points in the loop, we can choose what to do and organize context in different ways. So with LangChain TypeSafe, this is something that if you go and install LangChain TypeSafe, you can take this straight to boot.

21:41 I specify auto mode middleware. This is saying in Claude and Codex, this sort of behavior is called auto mode. We specify some tools, the two tools that we have up here. Specify some instructions saying if this action is going to be dangerous or not. We set some of those criteria that Allie mentioned, kind of went as the true and false of when we want these tools to be rejected or not. And then let me run this real quick. And then if I run this here, this is a pretty safe result.

22:14 customer, yeah, C123, add the internal notes. Pretty safe actions, we can go ahead and run it. Yeah, okay, CRM note added. This was fine, the status was success. If we take something like a destructive call, if we say, I want to delete this customer, I don’t want confirmation, if I run this, Then at this point, Jev, right before the tool call gets made,

22:46 is going to decide if it’s risky or not, according to the criteria that we have up here. And then status error, yep, it was classified as risky, so we didn’t wanna execute the tool. So we lost– - That’s just like– - In share, by the way. - Sorry about that. - It’s just a moment ago. - Cool. Yeah, anyways, you could see that it aired out. It was classified as risky, so we didn’t execute it.

23:17 Another case that we like, so yeah, this is pretty exciting because before the way that we had to do this was make another LLM call, which was pretty expensive. It was like very latent just because model had to reason over the tool call and yeah, yeah, the implementation was pretty naive, but with Jev makes it a lot simpler. A second kind of case that we pretty immediately latched on to was this idea of model routing. So depending on the task, I know models are going to be good for certain things.

23:48 I don’t want to slam Astra on every single task that’s possible. So we could specify a fast model or a more in-depth model. So very similar before, this is another middleware that you can export from. >> We lost your screen sharing again. >> Interesting. >> I’m just being the chat. >> Yeah. No, I appreciate it. I appreciate it. Let’s try this. This might be a little bit better.

24:19 Does that work? >> Yeah. We got it. >> Cool. Yeah. Specify the router in the same way. Yeah, it’s specified the instructions for when we should use different models. We’re going to try running this one more time. So I can run this, rewrite, like super basic construction. We’re going to use a cheap frontier model for it. Yep, it selected the fast route.

24:50 So you can see that it’s opted to use Luna for this specific case. Jev made that decision and it was super fast because of it. We can also do this one. This is probably going to take a while because Sol’s going to go and do Sol things, but I mean, it’s like, yeah, we can intuit that. like it’s gonna take a while to execute. That means that, you know, so we’ll pick this path. And the other interesting thing to come out of this is with this, we can, because we’re using this middleware,

25:20 we get tracing into the system out of the box. So we can see exactly what’s happening. We can see all the decisions it’s making. I think this is one of the, yeah, so this was the model router middleware that we saw. So I look in, I can see that before the agent runs, that’s what this before agent thing is doing. We’re running a classifier call against Jev and determining what the output is. So we pass in some details about the different models

25:52 that Jev can choose from, pass in some of the state with the initial query and things like that. And then in the output, we can see that it shows fast, which means that we opted to use Luna for this case. - And one interesting thing that Hunter is showcasing here is like, you know, with LLMs, I think it’s pretty common knowledge that you want kind of observability into what they’re doing, right? Like LLMs are sort of a black box. You want to be able to tell, you know, what were my inputs? What were my outputs?

26:22 Can I score? How, you know, evaluate how my agentic applications are doing and, you know, change my configuration for my agent to make it perform better and observability and evals are still like very important for even though like decision model class that Jev is. And so LangSmith is one great way to kind of visualize your state and state out and score how that’s doing for your use cases. - Yep. - So I really like you showing that observability

26:54 because one of the like top tips that I give to people when they start building with Jev is to, like no matter what system you’re in, build a way to get auditability of the actual TypeSafe calls. Because the most likely, especially if you’re like vibe coding to start, which a lot of people that’s like how they’re doing it, the models out there are pretty good at like understanding how to think about like system one

27:26 questions and kind of like compose them together. We have found that they are not typically very good at actually writing the instructions and the criteria. We think that’ll get better with time as they become more trained on more projects that use like Jev and System 1 models. But I like to give two tips, which is like make sure that you have a way that you can like inspect all of the calls that happen to Jev so you can like look at what the answers are like as you’re debugging and figuring out like what’s working. And then the second is something that I forgot.

27:57 Oh, especially if you’re vibe coding, tell your agent to put all of the copy, all of the instructions, criteria, anything that’s being fed into those into one centralized place, or someplace that you can easily go in and tweak it. ’Cause that’s gonna be, if nothing else gets a human touch in your code base and you don’t open up a single other code file, you are gonna wanna look at that one. Because you’re gonna be able to get huge performance, like an accuracy increases by just taking some questions

28:29 that sometimes are written very poorly and just making them more sensible. - Yeah, thank you, Hunter, for the demo and Allie for the tips. I think this is just a very new type of model and so everyone’s gonna need to slowly learn how we can best develop with it. One interesting resource that I stumbled upon was this resource called Jevify. It’s just basically a skill. I’ll paste the link in the chat. And this is like a prompt that you can give a coding agent to help it look over and scan your agent code

29:03 and figure out like, where am I using an LLM for decisions that actually would be much better with a decision model and how can we plug that in? How can we architect that to save on latency and cost? And then also importantly, like, where is an LLM the right tool here? And like, we need to leave that. So excited about that. I think it would be great if we could– - I saw that specific project and love it. Like we wanted to, if we had had like more time leading up to launch, we talked a lot about creating

29:33 like a brainstorming skill, which basically is exactly what the person has created. Like because there’s oftentimes like low hanging fruit, especially in like an LLM integrated piece of software. There’s usually low hanging fruit for Jev to be like a swap in and then sometimes less obvious things. And so the Jevify skill there is pretty darn cool. is there one more thing I can show, ’cause I made some promises and this landed literally five minutes ago inside of LangSmith. I just want to, sorry to hijack real quick,

30:04 but this is the thing that I love about working at startups is that we can ship so quick. We have nice rendering now, so you can go in and actually interpret some of the things that are inside of your LangSmith trace. So you saw it before that it was like pretty ugly [audio drops out] >> Yeah, those are working at LinkedIn.

30:36 Awesome. Well, we’re going to save some time at the end for community questions, but Allie, we have a couple of questions for you first, both that we’ve seen on Twitter and just internally from LangChain, Jev Power Users. So one thing that we’re curious about, I believe the context length for Jev is 32K tokens right now. Is that correct? It’s basically, it’s slightly complicated. It’s 32K for the state plus your largest question.

31:11 So like if you imagine a state and the question together is 32K and then there’s around a 64K cap for the state plus all of the questions together. That’s just when you would need to like separate it into multiple calls. - Okay, nice. Is there, do you guys have plans to like extend that context length or support other types of content, like multimodal, what’s your roadmap like there? - So our official answer to multimodal do we support images is not yet, with no further expanding upon that vague

31:45 promise. And I would also give the same answer to things like increased context length. We also want to go faster, cheaper, like this is the beginning. And if you read like the launch blog post on our website, like written by Diogo, it really does explain like where we’re at in this. We see this as like the beginning of this revolution that we that we want to help, you know, drive forward. And so yes, improving in every angle is what we expect. But no, I’m not not putting any specifics

32:18 or promises on timing or any of that. But follow us because we’re we’re we’re moving fast. Yeah, yeah, that’s great. I saw also Diogo put out some cool content, I think yesterday about like, how to think about Jev for coding agents. And like, how it can plug into some of those pieces, it seems like, interesting use cases that were highlighted were along the dimensions of like progressive disclosure and tool selection. And then also, like RLMs and code mode seemed

32:51 like an interesting case. Are there any other really compelling use cases that you’ve seen or like Jev in production that’s interesting for folks to know about? It’s like it’s so it’s so vast like what people are doing and they’re moving so fast. I want to be like let me just open my Twitter feed and like look at the top 10 posts. I bet all of them will blow me away at this point, right? And that’s the most exciting thing about where we’re at right right now is like, we knew the community would take a lot of these things and run with it.

33:24 And we are seeing that. You already talked about some of the, I think, what I’d call low hanging fruit, like using it for safety classification of tool calls, model routing, especially up front where you’re like, oh, is this overall question, does it require this, that, the other thing. I, my personal like biggest excitement is the projects that I see that are in the real time,

33:57 real time use cases. And real time was like a, it was a big like aha moment for us internally because we always called like, we knew that we were very fast and we knew that we were very cheap. And then all of a sudden it was like, I had an epiphany one night, I was like, wait, when you’re fast and cheap that unlocks real time. Like if you’re too expensive, trying to use something in real time, it’ll just become cost prohibitive. And if you’re too slow, you just can’t keep up with real time. But real time means that like real time chat filtering to me, I made a demo for this early on, but we didn’t end up using it for our launch demos.

34:29 And now people have already made better versions of like live filtering, a massive streaming Twitch stream of, tons and tons and tons of chat when you get these channels that have got like a hundred thousand viewers, right? You can filter those live And it only around 100, 150 millisecond delay in making it appear, it feels like real time. Like it doesn’t even matter. There’s like more latency than that just from like Twitch’s like displaying it, right? So you can, you can, you can,

35:00 because you can score them along an axis, you can even make it where the user can just like move like a scrubber of like how filtered do you want it? Do you want only the most like productive, like interesting questions being asked or do you want all the emoji spam, right? And you can just like literally just like turn a dial. And like that’s just like so cool to be able to do that. And in that parallel call, you can also classify them as questions or statements or hype messages like all at the same time, all in that 100 milliseconds for every single message and it won’t break the bank.

35:32 So yeah, I just, there’s so much. That’s just like the first one that came to my head. - I think one of the, I am glad you mentioned the real time element of it ’cause I think in like one of the launch posts, I saw the demo you guys did where you got Jev to play Doom real time. That was like one of those moments that was like, this is incredible, things are so cheap and fast now, it’s gonna open up like a whole new paradigm. - Yeah, that was my baby. The Doom demo has been my favorite thing to work on the last couple of months here. - Nice, well yeah, it blew me away.

36:04 I think it blew a lot of people away. So props to you, props to the whole team for building it. - Yeah, I agree. I think Twitter has really done something else with Jev over the past six days or so. I feel like I’ve learned about a new use case every five minutes. Are there any other interesting, before we pivot to the community questions, any other roadmap items that you wanna share? I know faster and cheaper is,

36:35 Like obviously, this is a lot of work on neural Zen. - Not, I would say not specifically, you know? It’s like a watch this space. You never know what next week will be with like how much incredible demand there has been. Right like in this immediate moment, we are like working on increasing capacity to get more people in. At least at the start of this, our signups were still paused as we managed demand and make sure that we have like high quality service

37:19 [audio drops out] [audio drops out] - Oh, not a lot. Not a lot. It’s been wild and crazy. And everyone is working like incredibly hard. But it’s also so many smiles. It’s so much smiles here, like people are excited. The energy is just electric in the office. - Amazing. Well, with that, let’s pivot to a couple of the questions from folks on the webinar.

37:50 the most upvoted one right now is, is Jev trained on the same data as a typical LLM? - So the, so it is, it is, sorry, I’m like reading my, reading my notes on this one, ’cause I don’t do, I’m like not an ML person, and I’m not involved in like the actual training and stuff. We do use a mixture of pre-trained models to train ours. our data that we use to train our model

38:22 is all synthetic data created in-house. And that’s part of our secret sauce. But it goes far beyond, the process goes far beyond post-training. And there’s a lot of hand-waviness in there because we do have breakthroughs, which is part of why we’re able to create what we’ve created for it to be this amazing. And so we’re going to hold on to some of that secret sauce for now. - Nice. I’m kind of aggregating a theme that I’m seeing across

38:53 some of these questions, which is like, how do you kind of tailor the either like confidence or probabilities that you set in your code as thresholds for like, okay, if the confidence is above X, then we’ll do this or if the probability is above X, is that, yeah, it seems pretty domain specific. How do you think about that? - I would say that’s exactly right, is that it is domain specific. My, I think it was written like multiple places in the like tips in our documentation where it’s like,

39:23 you should test your domain, like test your actual like data and use case to make sure that you under like, first of all, make sure that it’s like in distribution, make sure that like, “Jev is good for your use case, “don’t blindly shove it into production without testing.” But also like, that’s where you should be looking at like what the thresholds are. you know, if you’re finding yourself setting a threshold like super low, like especially like below 50% for Noul, because that almost certainly means that either there’s a problem and like this is like an aspect

39:54 of our model jaggedness where maybe we just like aren’t good at that particular use case yet. Or it means that maybe your questions need some significant tweaking. But so it is about testing it. And then I will also say that a big thing about using Jev effectively is to, like confidence thresholds is only one way to use the probabilities. What I think is really important to know

40:25 is that the confidence that’s returned by our API, that is a computation, is like a statistics computation on the probability map that you get back. It is a deterministic computation that you could do yourself. That is not a silver bullet. It does not, it represents in, in I think primary use cases, it’s a good analog for confidence. But that is the cases where you are expecting like a single dominant answer.

40:55 The confidence is mostly a measure of how much is there like disagreement, how much is there a distributed spread in the probability map. And you could have use cases where you actually expect two or three of the possible options to be competing for top probability, and in which case your confidence is gonna be like super, super, super low, because that’s the confidence in there being a single dominant answer. But in some of these use cases, that’s actually normal and okay. And so I also find that coding agents, they really like to see that confidence and be like,

41:25 yeah, we should always do a different thing if confidence is low. That’s not always the case. Like I built a shopping cart. Actually, the Doom demo is a great example. The Doom demo says, which monster, where should we be looking each frame, right? Like this monster, that monster, should we be looking at that item, et cetera, et cetera. None of those is like, oh yeah, that one’s gonna be like 100% and everything else is gonna be 1%. No, no, no, there’s like actually a lot of solid options. And so we just take whatever one is the top choice.

41:56 You could even do some fancy things, like adding a bit of like decay where you make the most recently picked choice stickier. and so it requires more of a swing to change it. Like that’s an interesting pattern I think to play with. But if we were to like naively, and what some of the first like vibed up versions of the Doom demo brains used confidence gating, it was like the agent was like, oh, confidence is low on this, so I should just fall back to like looking forward or something, it’s like, no, no, no, no, no, no, it’s like a bad use of confidence because it just doesn’t apply in the situation

42:26 because we expect there to be competing answers. We just wanna pick the best one. So that’s like just one of many potential examples you don’t want to just blindly trust confidence to work in your particular use case. It’s a good shortcut for a lot of simple use cases, but like always test, always check and be creative. There could be other probability math. Sometimes you just want like the top probability, raw percentage that you’d wanna threshold on. Sometimes it might be the top probability minus the second highest, or divided by the second highest probability gives you a different thing. Like all of these are different like

42:59 statistics math approaches to how to deal with it. >> Yeah, that’s super interesting. Anecdotally, I was working on a pretty naive application of Jev just using it to help me add labels to new GitHub issues so that we could automate more of our triage. I was perplexed by some of the probabilities and confidences that I was getting back in some of the results and was like, “I need to dig more into how to properly understand this for this use case.” Just calibrate myself as the developer.

43:31 Yeah, I think so maybe time for one more question. I think this one is interesting It says as far as I understand Jev’s status input token limit is yeah around the 32k that we talked about Are there any decision degradation? Consequences on large inputs. How should we think about that for Jev? It’s a really good question We have a a doc page in our docs called Model Jaggedness. And I highly recommend checking it out.

44:02 It is our, this is a part of our, like one of our like cornerstones of our company culture and like our brand is honesty. Like we don’t, we’re not, we don’t wanna say Jev is better at anything that it actually is, right? We are not an over promise culture here. And so the Jaggedness article is like our list of like known ways that we know that the model doesn’t do great in. And I believe, as I’ve read so much things recently, so I could be wrong, but I’m almost positive that one

44:33 of the things in there is talking about long contexts and that it can degrade accuracy if you put in a huge amount of state. I would say the guideline for that is make sure you’re only providing in the state what is actually necessary for making the determination. You don’t wanna just like throw the whole world at it and then ask it to make a determination about a part that is like just a tiny slice of it. Now, sometimes you don’t have any other choice, right?

45:04 Sometimes you truly are evaluating a large unstructured like wall of text and you can’t trim that down in code. That’s why you’re trying to use an AI model to like to evaluate in which case, iterate on questions, figure out like what works best, test it. But in a lot of cases, we’re constructing the state from multiple bits of data that we have in code. And so that comes down to the system designer to you as the programmer to consider, is this part actually needed? Do you need to add that to the state? Do you need to add this to the state?

45:34 It’s also better, more cost effective to not throw more tokens at something you don’t need. Jev is so cheap that most of the time you can just throw extra at it and not care. But you should be mindful about what you’re including and definitely test that. That’s like a big thing. And also in the Doom demo, like the state, I played a lot with like how much detail to provide in the state. And at one point I had like a recent actions array. This is like everything that happened recently, but it was so granular. It was like every like bullet of damage and stuff

46:06 that it would aid up a huge amount of the state. And it did cause the like the overall intelligence to drop a little. So I was like, okay, we should like clean that up. That’s like kind of a pointless amount of detail. So there is like good tweaking to work with that. That being said, every single thing on the Model Jaggedness page is things that we think could get even better in the future. Yeah. Yeah, I really appreciate that transparency. And I think it’s great thinking back to the Jevify stuff. It’s like, where should we use Jev? Where should we not? It’s not an answer for all use cases,

46:39 but can be really great for some. And I think it feels very full circle to end this conversation talking about basically context engineering for Jev. We have come up with many names for that in the past. In fact, we’ve run webinars on harness engineering, on loop engineering, on filling the, you know, now we’re talking about Jev engineering maybe, but yeah, I think it’s still important to think about like, this is still a model, like how can we optimize what goes in in order to optimize what comes out.

47:09 And with that, I just wanted to say, thank you so much, Allie and Hunter, for joining for this almost emergency webinar on building a harness with Jev. We are so excited to see the enthusiasm around the Jev launch. We’re very bullish on using it and other decision models internally. And yeah, thank you so much for joining us. And thanks folks for joining in your good questions. - Thank you. And I will also take an opportunity to plug our Discord. We wanna see what you’re building. We wanna see your cool projects. If Discord’s not your thing,

47:40 you can also like tweet it to us on X. That’s also cool, but like a lot of people showing a lot of really cool stuff on our Discord and a lot of our developers are engaging with the community in there too so we’d love to see you. I can plus one the Discord it’s a pretty cool place I’ve been I’ve been lurking in there for a little bit. Yeah. Okay.