CLM-8B vs Jev: What the 9× Speed Demo Actually Shows

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: CLM-8B vs Jev: What the 9× Speed Demo Actually Shows · Channel: Prompt Engineer 48 · 08:56 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models

Summary

CLM is an open contrastive model designed to rank a set of candidate actions, not to generate answers or replace a reasoning model. The presenter ran a small tool-routing test and found sensible choices in straightforward cases, but CLM chose an irrelevant option when the correct tool was missing, showing the need for an explicit abstain path and guardrails. Its speed advantage comes partly from separately encoding states and actions, allowing repeated candidate actions to be cached, though each new state still requires the encoder. The team’s benchmarks show faster results than Jev on the displayed tasks, but performance varies, some game results involve a safety shield, and the coding scores use a separate verifier setup.

Key points

CLM scores candidate actions for a given state; it is intended for decision points in a larger system, not for drafting or solving tasks by itself.
The downloadable projection head is about 75 MB, but running it still requires the QN 3 8B encoder.
In the presenter’s small tool-routing test, CLM selected sensible options for a code-fix request and a request to find recent information.
When the needed code-editing action was removed, CLM still ranked web search first rather than abstaining; an abstain option must be designed and tested.
CLM encodes states and actions separately, making it possible to cache action vectors when the candidate set is repeated.
The approach is a better fit for repeated decisions over a known menu than for choosing among unique, long plans.
The team describes three training stages: broad question-answer alignment, hard negatives, and agent trajectories.
The team’s chart shows CLM faster than Jev on all four displayed tasks, but Jev scores higher on BFCL tool calling and Wiki Racing.
In the dinosaur-game results, both models survived five courses with a safety shield enabled; the report says CLM agreed with the physics planner about 66% of the time and Jev about 99%.
The reported coding results use a separate setup in which a fine-tuned CLM head verifies candidate solutions; they are not zero-shot results for the release general head.

Chapters

Start	Chapter	Summary
00:00	What the speed demo does—and does not—show	The presenter introduces CLM and Jev, then frames the evaluation around whether CLM makes useful decisions and notices when the right option is absent.
01:03	Running CLM and testing tool routing	The presenter explains the model setup and tests CLM on routing examples, including a case where the needed action is missing and CLM does not abstain.
04:09	Why the architecture can be fast	The video explains separate state and action encoding, caching, the limits of reuse, and the team’s staged training approach.
06:16	Reading the benchmark results	The presenter compares the team’s speed and task results, notes the safety shield in the game tests, and distinguishes the coding evaluation setup.
08:24	Where CLM may fit	The presenter sees potential for bounded routing and selection tasks, while recommending guardrails, evaluation on real data, and a generative model for other work.

Quotes

It returns a score or probability for each action. — 01:03

If you want an abstain route, you must design one and test it. — 03:39

that is the source of the architectural speed advantage. — 04:41

I cannot use five clean survivals as proof that the two models made equally good independent decisions. — 07:20

CLM is not a new chat model. Trying to answer everything. — 08:56

Concepts and entities

CLM 8B, Jev, TypeSafe, Qwen3 8B, RunPod, aprendizado contrastivo, Safety shield, BFCL, Apache License 2.0

Full transcript (original language)

00:00 So look at this. On the top, a model called CLM is playing the dinosaur game. On the bottom is Jev, the new decision model from type safe. The CLM site is answering in about 16 milliseconds. Jev is around 150, that is roughly nine times faster in this run. But here’s the question I actually care about. If I give this model a decision in my own agent, will it pick the right action? And when the right action is missing, will it notice? I downloaded the code, ran the released model on a runpod GPU,

00:32 and tested exactly that. Then I went through the numbers behind this launch because the demo has a detail that changes how I read it. First, what even is this thing? CLM stands for contrastive language model. The team calls it a system one model. Type safe used the same label when it launched Jeff a few days earlier. The idea is simple. An agent often does not need another paragraph of text. It needs to choose a tool, route a ticket, rank several candidate fixes, or score a possible next move. You give CLM the

01:03 current state and a set of actions. It returns a score or probability for each action. It does not write a new answer from scratch. So, I would not ask this to draft an email or solve a coding problem by itself. I would put it at a decision point inside a larger system. That distinction matters for the headline. 8 billion refers to the Quinn 3 8 billion parameter encoder under the hood. The downloadable CLM projection head is only about 75 megabytes, but you still need to run the QN encoder to produce the embeddings. This is not a 75

01:33 megabyte standalone model. So, what I did is clone the public repository, start Quen 38B in its pooling mode, and run the CLM server with the release checkpoint. I used a temporary A40 on RunPod because the 8 GB GPU in my laptop is too small for this reference setup. The server exposes a little playground, but I also sent my own request directly to its API. So, I could save the exact input and output. This is not a reproduction of the team’s benchmark. It is a small smoke test for a real builder use case tool routing. I also ask the

02:05 same questions of CLM raw, which scores with the underlying encoder embeddings without the trained CLM head that gives me a basic ablation on the same machine. It is not Jev and I’m not going to pretend that it is. Here is the first state. ACI job is failing because piest cannot import a module after a code change. The user wants the test fixed. I offered four actions. Edit code, search the web, generate an image, or open the calendar. CLM chose the code editor with 98% of its probability on that option.

02:36 The raw encoder also chose the code editor, but with a much flatter distribution, about 78%. Then I changed the state. Now the user is asking about a model launched this morning and once the announcement source CLM chose web search which is what I would do too. It gave web search about 70% raw embeddings also picked it at about 46%. Again this is one request not a reliability score but it shows how you can use the trained head to sharpen a bounded routing decision. My third question was less

03:08 clean. Look at the new CLM paper and explain the code example in the repository. Should the agent read the paper first or inspect the code first? CLM picked the code editor very strongly. I would accept that as a reasonable first step, but that 99% tells us more about this candidate set and these descriptions than about some universal confidence that the whole task is solved. Now for the failure case, I ask it to fix a failing Python test but deliberately remove the code editor from the choices. The only options left were web search, image generation, and

03:39 calendar. CLM still picked one web search with 84%. It did not say none of these is the tool I need. That is not a software bug in the API. It is the contract. Give it candidates and it ranks them. If you want an abstain route, you must design one and test it. Do not turn these probabilities into permission for an agent to take any action automatically. You can find every prompt and response from this small test in the project files. I did not have Jev API access, so my own run does not establish the nine times comparison. That speed figure comes from the team’s

04:09 published evaluation. So why should this approach be quick? The important design choice is that the state and the actions are encoded separately. CLM takes the current state through one encoder path. It takes each candidate action through another. It projects them into the same space and checks which vectors line up best. The model was trained to pull a real state action pair together and push mismatched pairs apart. The objective is called contrastive learning for an agent

04:41 with the same action list on every step. The action vectors can be computed once and cached. In the Mario demo, there are four fixed actions. The game screen keeps changing, but the action descriptions do not. So, after setup, each new step only needs a fresh state embedding and a few similarity scores that is the source of the architectural speed advantage. The repository even has a separate vector cache for repeated states and actions. There is a limit though. A new state still has to pass through Quen 38B. And if your candidate

05:12 actions are different every time, there is less to reuse. This is a good fit for repeated decisions over a known menu. It is a less obvious fit when every action is a long unique plan. The team did not just bolt a classifier onto QN and call it done. They described three training stages. First, around 60 million neatron question answer pairs teach broad alignment between a question and its correct answer. Second, around 30 million hard negative examples make the model distinguish plausible wrong

05:44 answers from the right one. Third, around 1 million agent trajectories adapted to states and actions inside actual workflows. Here’s a detail I liked. In the team’s experiment, pre-training alone picked the correct answer against 10 hard negatives. about 52% of the time. A later hard negative stage lifted that to about 69%. Starting with hard negatives from the beginning peaked around 62% at the same compute budget. They also replayed the broad question answer data during agentic post training so the head would

06:16 not forget it. Those are team experiments but they explain why this is more than ordinary embedding similarity. The blog also shows scaling curves for compute data head size and encoder size. Their strongest gain came from increasing the encoder. I would treat those as evidence for this training family, not a promise that the next model automatically solves the harder tasks. Let us come back to the headline on the team’s zero shot chart. C0M is faster in all four shown tasks. In T-Rex, it is 16.5 milliseconds versus

06:48 149.8 for Jev. In Mario, about 33 versus 133. But comparable performance is not identical performance. Jev scores higher on the BFCL tool calling result. And in Wiki Racing, the team’s chart has CLM at 26 out of 30, while Jev gets 30 out of 30. The sample sizes in the game demos are small. The dinosaur run is especially interesting. I opened the JSON results in the repository. Both models survived all five seated courses, but the safety shield was enabled. The harness already labels safe and unsafe

07:20 moves and can override a model choice. Its report shows CLM agreed with the physics planner’s best move about 66% of the time, while Jev agreed about 99%. It also records thousands of shield interventions in the CLM run. So, I can show you the speed difference in the video, but I cannot use five clean survivals as proof that the two models made equally good independent decisions. The bigger coding numbers are a different setup entirely. The team reports 81.6% on 38 held out deep SWE tasks and 87.6% 6% on 30 held out

07:51 terminal bench 2.1 tasks in each task. Another strong model first generated several candidate solutions. A fine-tuned CLM head then acted as the verifier and picked one. These are not zero shot scores for the release general head and they are not full public leaderboard runs. The release repository does provide a checkpoint and evaluation path for at least the deepsw result which is good for scrutiny. I would still like to see independent reruns before calling it a settled state-of-the-art. So would I use CLM today for a tool router ticket classifier or best of end selector with

08:24 well-defined options? Yes, it is interesting enough to test. The code and head are open under Apache 2.0. The interface is simple and the caching architecture is a real engineering idea. My small live run made sensible choices in the easy cases, but I would put guard rails around it. Include a real none of the above path if that can happen. Measure mistakes on your own data. Do not interpret a high softmax probability as proof that an absent option is somehow safe and keep it generative or reasoning model available for the work. This score does not do that to me is the

08:56 useful story here. CLM is not a new chat model. Trying to answer everything. It is an open attempt to make one narrow part of an agent loop faster. Deciding among actions, the team’s speed charts make that worth attention. The conditions behind those charts tell us exactly where to test next. If you’re building an agent with a repeated action set, tell me the decision you would give CLM. I want to see where this design helps in a real workflow and where the candidate list becomes the bottleneck.