Decision Model Showdown: CLM vs Laya vs OpenJev vs Kev vs Jev

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Decision Model Showdown: CLM vs Laya vs OpenJev vs Kev vs Jev · Channel: Fahd Mirza · 08:19 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models, AI safety

Summary

The video compares five decision models—CLM, LLaMA/Laya, Open JEv, GEv/Kev, and JEv—on customer-support routing and a social-engineering scenario. In the customer test, all five selected Billing, but differed in confidence, frustration ratings, and speed; the presenter highlights CLM’s local speed and confidence while noting weaker results from some alternatives. The second test shows that only CLM and JEv flagged a request for emergency production data as a critical risk, while the other models recommended granting access. The presenter’s main takeaway is to test decision models on the actual use case before trusting them in production.

Key points

The models are tested with the same angry-customer scenario and asked whether it is urgent, which department should handle it, and how frustrated the customer is.
All five models identify Billing as the correct department.
CLM 8B reports 97.9% confidence, labels the customer very angry, and completes the task locally in 24 milliseconds.
GEv/Kev gives a 94.2% Billing result and describes the customer as frustrated rather than very angry.
Laya gets the answers directionally right but has the lowest department confidence, at 88.6%; the presenter notes it has 421 million parameters.
Open JEv selects Billing with 54.7% confidence, which the presenter describes as close to a coin flip.
The presenter says JEv is the only closed model in the lineup and the only one that costs money; four are open source and can run locally.
In the social-engineering test, CLM and JEv flag the request as a critical risk and recommend escalation.
GEv/Kev, Laya, and Open JEv recommend granting access in the security test, despite the unverified approval and urgency claim.
The presenter advises testing models on the intended use case before using them in production.

Chapters

Start	Chapter	Summary
00:01	The models and the customer-support test	The presenter introduces decision models, sets up five terminals, and explains the shared customer-support questions.
02:39	Customer-support results	All five models route the case to Billing, while their confidence, frustration ratings, and speed differ.
05:13	Model lineup and benchmarks	The presenter compares the models’ openness, deployment options, and reported benchmark focus.
06:15	Social-engineering test and conclusion	CLM and JEv flag the risky access request, while the other three models recommend granting access, prompting a warning to test models before production use.

Quotes

They just decide. — 00:01

Every single model got the department right. Billing across the board. — 02:39

it did all of that in 24 millisecond running locally on my GPU. — 03:40

none of them spotted the manipulation. — 07:19

Test your models on your actual use case before you trust them in production. — 08:19

Concepts and entities

Decision Models, CLM, Laya, OpenJev, Kev, Jev, Fahd Mirza, Stanford, NVIDIA, Vast Compute, DeepSeek, Terminal-Bench, Jev Bench, TypeSafe

Full transcript (original language)

00:01 The AI world is exploding with a new category of models that really is rattling the scene at the moment. These models don’t write, they don’t think out loud, they don’t generate a single token. They just decide. Fast, calibrated, instant decisions with probabilities you can actually trust in production code. We already have covered few of these on the channel, but for the purpose of these videos, we are running five of

00:31 them head-to-head on the same task. Same question, same angry customer. CLM, LLaMA, Open JEv, GEv, and JEv. Let’s get started with it. This is Fahad Mirza and I welcome you to the channel. Recently, I also have started this free weekly AI newsletter which you can subscribe at fahadmirza.substack.com. Let me take you straight to my terminal and you can see that I have opened five terminals. This is for LLaMA, Open JEv,

01:04 GEv, JEv, and CLM. As I said, if you want to know how to get them installed, you can check out my channel. I have covered almost every one of them and they’re very similar to install. The code which I’m going to use is this. And similar script for each model you can see on the left. So, this is what we are testing. One angry customer who got charged twice and cannot reach anyone. We are asking every model three things. Is this urgent?

01:35 Which department should handle it? And how frustrated is this customer? Same state, same questions, all five models. Let’s run them one by one in terminal and see what they say. I’m just going to first start with CLM. It was very quick. I’m going to now start with Jeff. It was also very quick. I’m going to go with Kev. While it runs, open Jeff and then Laya.

02:08 I will wait for all of them to come back and we will compare these. Some of them are running locally as you can see on my system. If you’re interested, I will also show you my VRAM consumption. You can also use their API. So, all in all on my 48 GB of VRAM, they all are consuming under 30 gig of VRAM. Um and if you want to know individual, just search my channel for individual videos of these models. If you’re looking to rent a GPU on very good price, you can find the link to Vast

02:39 Compute in video’s description with a discount coupon code in um the description again. So, please do check them out. So, every model has come back with the result. Let me quickly browse through. So, this is all of these five models result as you can see on your screen. Just to make things easier and compare properly, I have just created a dashboard out of this. So, look at this. Every single model got the department right. Billing across the board.

03:09 That really tells you that this category of model works. But, the differences are in the confidence and speed. So, you see if I’ll just go down a bit and let me make it a bit bigger. Look at this. Sorry. Look at this now. You see, Jeff 1.13 came in at 100% confidence. On billing and called the frustration as very angry. The proprietary training is showing here by the way. But, you’re paying per token and locked into their

03:40 cloud. CLM 8 billion is really the good one. 97.9% confidence, very angry at perfect, two out of two, and it did all of that in 24 millisecond running locally on my GPU. You can also run it on your CPU. We saw that in our video. That is six times faster than Jave on their own cloud. Open source Apache 2 runs on your own hardware. Gave 4 billion is solid and known as 94.2%

04:10 on billing and it called it frustrated rather than very angry, which is actually the most calibrated read of the three stories and scores so far. Closest to the numbers in my opinion. Laya got everything right directionally but with the lowest confidence on department at 88.6% and also called it frustrated. That is why I don’t think that is that impressive. But remember this is just a 421 million parameter model, the smallest here by far and it is still in the game. But again, if it

04:42 is a production use case at this point in time, I’m not going to recommend that. Open Jave got the right answer but barely 54.7% on billing is almost a coin flip between two options. This is a zero shot NL one model not specifically trained for typed decisions and it shows. Fine tuning and number will look differently in my humble opinion. So all in all these are the results of our test which I just showed you. If you want to see the head-to-head comparison, this is again a proper

05:13 showdown about what exactly these models are. So if you look at this lineup, you can see that two are from Stanford and Nvidia, one from a solo developer, one from Jared Palmer who built a Formic and one from a funded startup. Four fully open source, four you can run locally on your own hardware with zero data leaving your machine. Jave is the only closed one here and the only one that costs money. At this point in time, this whole category barely existed a month ago. But

05:45 remember, classifiers are not a new thing. We already have discussed it in our other video. I’m not going to go into that detail. But look at this uh published numbers. Uh and this is where it gets really interesting. CLM is the only one reporting Deep Sea and Terminal Bench because it was built specifically for agentic verification, picking the best solution out of many candidates. Jev leads Jev Bench, but that is Type Safe’s own benchmark, so factor that in. Open Jev and Kev are both competitive on

06:15 Jev Bench and they are open source. Liar skips all of this entirely because it was built, I think, for a different problem, multilingual support, routing, and calibrated decisions at scale, not agentic coding, different architectures, different goals, but the same core idea for each and every one of these. Let’s do another final but a hard test. So, you can see that in this one, I am asking that an employee demanding emergency production data with every success with unverified viable manager

06:47 approval and a $50,000 per hour urgency claim, a classic social engineering scenario. So, what I’m going to do here, I have already given that to these models one by one. They have provided me the result. If you look, let’s say, at CLM and Jev, they saw right through it. Both flagged it as a critical uh risk. Both said escalate and both were highly suspicious of social engineering at 70% and 93

07:19 um percent, respectively. These two passed the security test, right? But if you look at this Kev, Liar, um and even Open Jev, all fell for it. So, for example, look at this Kev. Kev said grant access at 84.8%, right? Quite confident one. Liar said grant at 85.9%. Open Jeff said 97.3% none of them spotted the manipulation.

07:49 So look, you can already see that CLM and Jeff both of them are the only two you would trust in a security sensitive genetic pipeline. The other three would have opened the door very very welcome uh ly. So please uh be aware of that. So look, decision models are powerful but they’re not all equal as you just saw. The same model that routes your customer support tickets perfectly might hand over your production database to the wrong person.

08:19 Test your models on your actual use case before you trust them in production. This is the real lesson from today. That’s it. If you want to help out the channel, please become a member by clicking on this link or in the first pin comment and please subscribe to this newsletter. Thank you for all the support. Highly highly appreciate.