Decision Models para Evals: Testei a API da OpenAI

Author: Ronnald Hawk | Engenharia de IA · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Decision Models para Evals: Testei a API da OpenAI · Channel: Ronnald Hawk | Engenharia de IA · 16:38 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Software engineering with AI

Summary

The video tests whether decision-model APIs can be used for evaluating a real AI agent, comparing them with the creator’s existing LLM-based evaluation process. It explains how evaluation questions, reference answers, business-informed criteria, and rubrics work, and why evaluations require calibration. In the experiment, the decision-model outputs could be adapted to the rubric, but results depended on thresholds and were not directly equivalent to the existing evaluator’s probabilities. The creator also compares cost, speed, and debugging value, concluding that a decision API is not automatically better and that engineering judgment and expertise still matter.

Key points

The experiment applies decision-model APIs to an evaluation of a real agent used by people regularly.
An evaluation compares an agent’s response with a reference answer using defined criteria and a rubric.
Evaluation criteria should be developed with business knowledge and stakeholders.
Evaluations need calibration; they are not a one-time setup.
The decision-model outputs required adaptation before they could be used with the existing evaluation rubric.
Predicate and score were tested, with results affected by the chosen thresholds.
The existing evaluator’s score is not the same as the probability returned by the decision API.
Prompt caching reduced the existing evaluation process’s cost in this experiment, so the decision API was not automatically cheaper.
The decision model provided less visibility into the reasoning behind its result, making debugging and investigation harder.
The video encourages engineers to understand their tools rather than follow hype or outsource their judgment.

Chapters

Start	Chapter	Summary
00:00	Decision models and the evaluation experiment	The creator introduces recent decision-model releases and explains the plan to test one on a real agent evaluation.
02:33	Comparing the APIs and their context	The video outlines similarities between the APIs, their image inputs, and why context size matters for evaluations.
04:05	How the evaluation and rubric work	The creator explains how an agent response is judged against a reference answer using criteria and a rubric informed by business knowledge.
08:15	Adapting and calibrating the decision-model outputs	The experiment requires calibration and changes to the rubric because the decision model returns outputs in a different format.
09:48	Results and thresholds	The creator compares evaluation outcomes across predicate and score settings, noting that thresholds affect results and are not directly interchangeable.
11:56	Cost, speed, and evaluation reasoning	The video weighs the APIs’ speed and cost against prompt caching and the existing evaluator’s ability to explain its scores.
14:31	Debugging, engineering judgment, and conclusion	The creator argues that decision-model results can be difficult to debug and urges viewers to build expertise rather than follow hype.

Quotes

Don’t outsource your minds, guys, it’s all you have. — 00:30

EVAL isn’t something you do once and it works. — 08:15

the Geval is not the same as the probability of exiting the uh decision API. — 11:24

This is investigative work, which is your job, it is indeed your own job. — 14:00

the more you outsource your brain to someone on the internet with “now it will work” and “now it’s the thing to do,” the worse it is for you. — 15:33

Concepts and entities

Decision Models, OpenAI, Cloudflare, Qwen, evals, rubrica, LLM as a judge, prompt caching, Rock Pro

Full transcript (original language)

00:00 This week was a terrible week for Jeff Bezos fans. That’s because OpenAI released the Decisions API based on an LLM, GBT6 Luna. And CloudFlir released their decision model. And this is even more interesting for our analysis here, because they explained how they did it on their blog, and they used the customized Quin 27B and Quin 9B to get that structured output and all that.

00:30 Why is this important? It reveals to us that the emperor has no clothes. As I showed in the previous video, we can adopt a new technology, but if we don’t understand how it works, we’ll always be swayed by hype, believing that a technology will replace our understanding. Don’t outsource your minds, guys, it’s all you have. But one of the use cases that could be interesting in Decision Modeling, and that I’d like to test, is EVALs. And in this video, I’m going to show you an

01:01 EVAL, not a hypothetical EVAL of a hypothetical agent that supposedly works. I’m going to show you a real EVAL from an agent that hundreds of people use every week, and I’m going to take what exists today, the technique I use today, and I’m going to take the decision model, apply the technique, and use it with it. And then throughout the video, we’ll see the results. And I hope you have the ability to analyze, and then, what do we do in engineering? We use the best tool for the job. We don’t become fans of A or B simply because that changes everything. Before

01:31 we get into the video, let’s talk about today’s sponsor. Rock Pro is the best engineering platform in Brazil. That’s where the professionals are. So let’s go, shall we? Decision model for EVALs. So I’m going to go through an experiment I did. It’s literally a study. And we’re going to see the conclusions we drew from this study. And what is G based on? It’s based on this guy here, this agent we have online. And that’s exactly the question we’re going to look at, okay? Okay, my agent remembers the current conversation, but misses important

02:03 facts, etc. Why am I showing this here? Because it’s literally in my EVA, okay? This is a question I created in EVAL. I’m going to show you what EVAL is. If you don’t know these terms, that’s okay . Uh, but this is what the real-world industry does, okay? This isn’t speculation. This is what I apply both at Rockpr and with my clients, and it is what the entire world is doing to improve their solutions, if you know what I mean? So, if you have a problem with your solution, it probably doesn’t work so well, it runs the risk of being

02:33 this too, okay? So, let’s go. Very well , we had the launch of Cloud Flir from Upai, as I mentioned here. And I recommend you take a look at the blog if you’re interested, but the important thing is that both APIs are very similar to the JV we knew, okay? However, OpenAI changed the name here; they called it “predicate.” The other two are the same, so this is just for your reference, okay? But the idea is exactly the same. You put something in

03:03 the context, and it outputs a response. The main difference between these two models is that they are multimodal, or rather, they accept images. So you can decide with images as well, which can be interesting, right? For your use case, the Cloudflare model has 64K of context, so you can send a slightly larger amount. If I’m not mistaken, the JV was 32, so it greatly limits the solution you can implement, but the OpenAI model is more interesting for EVAL because it has the context window like a normal model. So, the context

03:35 window is quite large, and we can explore that in EVAL. So, for EVAL, since the context is often large, it will analyze several things; it will analyze the user input, it will analyze the retrieved document in many cases. In this case, we’re going to evaluate a rule. It’s important that we have a slightly wider context window to see the result. Okay? So, that’s good. You already know what we’re going to do here. Let me explain how this works, okay? How does a judgment, how does an

04:05 EVAL work? You’ll have the response generated by the agent. So I came here, to Rockpr, right? There’s the question, there’s the response generated by the agent, and here’s its response, right? What it did. And we’ll have a reference of what the correct answer would be. So , if someone were to ask you this specific question, how should the answer look? So that’s the reference. And we’ll have a rubric that will evaluate it. If you think about giving this to a human to evaluate, you would have the rule, right? Ah, according to

04:36 this, it’s that and so on. For example, if someone is to evaluate the ENEM exam . So I imagine they have a rubric. So that’s exactly it. You send all this information to a judge, and they’ll say , oh no, this case passed, this case is so-so, this case is good, this case is bad. That’s basically what we’re talking about. So let’s go to our notebook to understand how this is configured. Let’s go, right? What do we have here? It’s exactly that question I showed you earlier, okay? My agent remembers the current conversation, but loses important facts when the translation changes, then the

05:07 summarization changes, okay? Uh, what should I study? So, this is a classic case of a student, a person trying to understand their problem and trying to find the material to solve it. And yes, we already have an answer here, okay? So, what I’m showing you is after the fact, right? After the study, okay? Okay? And this is the complete answer the agent gave, okay? Obviously, since we’re dealing with LLMs, the same question won’t produce exactly the same answer. So, what we do isn’t evaluate letter by letter, right? We evaluate

05:38 the overall context of the answer, and that’s why the criterion is important. You see here that he answered, the answer is quite long, everything’s great, right? What’s the criterion? How would it be resolved? It’s here, for this case, uh, the agent should look in my rule for phase two, etc., etc. So, the context that the judge has to know is this: Phase two, lesson 10. If my agent doesn’t bring this in his answer, he’s wrong. It doesn’t mean that if he brings something else too, he mentions this material and maybe this other

06:09 material here helps you, he’s wrong. OK ? So that’s the criterion I developed. And how do we generate this criterion? Uh, with business knowledge. For the following reason: since I was the one who recorded the class, and I created the agent, and I created the EVAL, it’s very easy for me because I know what I want to convey and I know what my agent should answer, because, at the same time that I’m a developer, I’m also a stakeholder. If you’re working in a company, this evaluation of the criteria for the user’s question, the

06:41 evaluation criteria, and the rubric are done in conjunction with the business. I recommend my previous video here, where I talk about how you should understand that your profession is entirely intertwined with the business. There is no AI and business, there is only one thing, okay? And the rubric goes exactly in that sense as well. Uh, you’re going to create a rubric, okay? This is the standard. The standard, okay? This isn’t something I’m making up, is it? It’s something I made up. You’re going to create a rubric that will evaluate. And you can do EVAL

07:12 without a rubric, but you’ll have difficulty standardizing it. So, it’s highly recommended that you make a note , okay? And then, uh, we’re going to start taking notes, okay? Uh, I use a technique called EVAL, which isn’t the topic of this video, okay? It’s a globally known technique, so it’s not an invention. And within this metric, I have value, okay? But I want to show you this here. Uh, when we’re doing EVAL, uh, it doesn’t mean the answer is

07:43 inadequate, it means it’s incorrect. Because, uh, in EVAL, many times you want to force a wrong answer to see if, uh, your EVAL is also correct. So, you uh test if it’s approved and test if it’s rejected as well. So, you have to keep that in mind, okay? And that’s exactly what we’re doing. Uh, here in our Bravissimo, let’s go back to our notebook, it’s in our Bravissimo EVAL, okay? So he did everything correctly, he failed, and we have a baseline. And here’s an interesting part about

08:15 decision models and EVAL itself, because since you have a probability of something being right or wrong, you need to calibrate. And this is one of the very important factors when we do EVAL. EVAL isn’t something you do once and it works. With EVAL, you’ll spend time calibrating. And this is extremely important for what I’m going to talk about from now on regarding decision models. To use the decision model, I need to make adjustments because, since it has a slightly different output than

08:45 the Gval output, I can’t just use my rubric as it is, my test hold as it is, to use the decision model. So, what I did in my experiment was try to adapt these outputs to a format that would be good enough for me to evaluate if the model would be good enough to use as an EVAL. The model isn’t ready to use EVAL directly, so we have to make an adaptation, and when I say EVAL, I mean EVA from LLMS, right? Very well. Okay.

09:17 And arriving at this conclusion, I believe that the two best opportunities here are in predicate and score. And that’s exactly what I did in the experiment. Let me show you, okay? Uh, and the first case, I had exactly a zero probability of it being correct, so the answer was rejected, which is what I expected. But here we lost a little nuance, right? And I don’t know exactly what happened. In the other case, the score is good because it gives a probability distribution, right ? So I transformed my rubric into each of these probabilities, and in the end,

09:48 I’m only adding the cases that would be approved. So in this case, it gave 8, and it was excluded because I was doing something with 5. That’s basically the experiment. So, I am able to apply my rubric or my EVAL to the IPI decision-making process; yes, I can. But now I’m going to show you the results. And then it gets interesting for this reason, okay? Uh, let’s go, right? Running the GVAL, okay? So, it’s exactly that question there, it’s

10:19 exactly that rubric, it’s exactly my judge, okay? It’s within my GVAL. And here I’m using a threshold of 07, so it’s 70%, if you want to think of it that way, and he passed. So, in all these rounds here, I passed this guy 15 times, right? 17 answers 15 times. And I ran this here several times. He passed all of them, okay? Yes, using predicate, okay? I used the same techniques and it worked. So, my portability is saying here, at least to

10:50 me, that this portability works, okay? So this is the experiment. It seems that the portability worked, which is something good for us, okay? Uh, but here in the score, what did I do, right ? Ah, okay. I used both the threshold 05 and 07, okay? Here in the predicate, okay? Why? Okay, this threshold here isn’t the same; it’s ported, as I explained. So, it’s impossible to know if it’s 05 or 07 FL 07. This is a matter of testing, and I strongly recommend that if you’re doing this, you test exhaustively, okay? Score, uh,

11:24 I had 15, 15, 17, great. But when I increased it to 07, I had an error, okay? Uh, it hit 0.68, which was very close, but uh, that Ket remains. Uh, in the end, what you have to understand is that the Geval is not the same as the probability of exiting the uh decision API. And this is very important for you to take into account. Now let’s show numbers. Numbers, numbers, numbers. You like numbers, right? Time. And then, you’ll say: “Man, it was worth it to

11:56 use the Decision API. It was much faster. It was much faster.” And then I can ask you a question. Do you need your EVAL to be very fast, or do you run it in batches from time to time? I don’t know. You answer me, okay? Uh, but the other piece of information I have here for you is that I’m using a normal model in GVAL, and I’m also using Luna, I forgot to mention that. So, I’m using Luna in GVAL, or I’m using Luna in the Decision API version, and in the other one there. What can I

12:27 do? I have a video here talking about prop casing. What did I do? I did prop casing of the input of my questions because the questions were repeating. I tested it several times. And what did that do? It made my final cost with GVAL lower than using the Decision API, okay? So, many times it’s skill that matters. Uh, you can do something that …It’s cheaper to use the existing model, giving you the answer you need,

12:59 it’s just using technique, right? So, if you’re an engineer, if you’re a professional, you should consider these things here. Again, it’s the aggregate cost of GVAL. And there’s something I want to show you here, it’s on my notebook, which is the following, let me see if I can find it here, okay? Uh, this whole thing here, open GVAL, okay? Yes, in GVAL I still have the following advantage. My model, my judge, right, when he uses the rubric, what will he

13:29 do? He will give the score, so he will give the probability distribution and will say why he gave that score. Why is this important? Because as I said, when you are creating a rubric, many times you need to understand what the judge’s reasoning was in that rubric for him to have given that score. This is because sometimes your rubric isn’t good the first time, and usually it isn’t, okay? And usually your judge’s criteria can fail. It can be bad. So you’ll start to understand that, okay, he arrived at

14:00 this grade using this criterion, this is the path he took. And then you can see, sometimes the problem is in the agent, but sometimes the problem is in the dataset you’re using to do the test , sometimes it’s in the rubric, and you’ll have to investigate all of that. This is investigative work, which is your job, it is indeed your own job. You have to be able to analyze this. And what’s the problem with using a decision model? Because the return is this, man. Okay? So, you start flying blind here, right? Because you can’t

14:31 know exactly the motivation, because it’s faster, so it doesn’t print a token. So, if it doesn’t print a token, it’s better. It depends. If you want to do something that you have mastery over and you understand what you’re doing, at least for Eval, it gets kind of complicated. But there’s a cool thing. If here, look, we can, uh, no, I think she’s not on the slide, right? If here we can, we can create a pairing, Suddenly it might be worth using, but then the work you’d be doing is like using a normal LLM, you’ll be able to

15:03 create your rubric very well, your dataset very well, great. Then you will perform this adaptation to use blind decision modeling; you are going to trust the number, my friend. The number that comes, you have no chance to debug . It’s like, man, it just comes. That’s it. So, I really believe that suddenly it doesn’t have that much value, okay? So, you come to your conclusion, as you see here, you don’t even have the justification for the lower cost, because I proved to you that using

15:33 promptcasting, I had a better return on my investment than using the Decision API, because I know how to use the API. That’s basically it. Things that I constantly show you here on YouTube, which I know is trendy, is about getting good at what you do. Uh, uh, it’s cheesy that you study, it’s cheesy , man. But that’s it, that’s all you have. Okay? Uh, the more you outsource your brain to someone on the internet with “now it will work” and “now it’s the thing to do,” the worse it is for

16:05 you. And the more you outsource your execution capacity to the token, the worse it is for you, because we’re reaching a point where whoever controls the tokens controls the world. But don’t let those who control the tokens and those who control the internet narrative control your mind. I hope this video has been useful to you. Uh, here I talked about a lot of things that can be advanced, so there’s no problem if you don’t know these things, but it’s necessary to show these things to the Brazilian air engineering community, because we need to get out of childhood. There’s a world out there

16:38 that needs people who want to do it, who can do it, who have the capacity to do it, but this demand isn’t being met because there are literally few people capable of doing it. And I hope that’s your desire. I hope that’s already your capacity. And we will conquer the world . It’s a matter of time. Uh, once again , no. There is competition. If you keep doing it, keep improving, keep getting better, there are many opportunities for those who are actually doing it.