Red Hat Tested Jev and the Results Are Awkward...
Read the Portuguese translation →
Author: Better Stack · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: Red Hat Tested Jev and the Results Are Awkward… · Channel: Better Stack · 02:37 · Watch on YouTube
Topics: AI agents, AI evaluation and benchmarks, AI models, AI safety
Summary
Section titled “Summary”Red Hat evaluated Type-Safe’s Jev decision model alongside eight other guardrails on prompt-injection detection and content safety. Jev won the content-safety test, but a small DeBERTa classifier running on a laptop CPU nearly matched a much larger model on prompt injection and was substantially faster. The results also showed that prompt engineering can affect models differently, and Jev’s measured latency included extra network time from calls between the UK and US. Red Hat concluded that decision models were not reliably faster, cheaper, or more accurate overall, while small classifiers remain hard to beat when labeled data and a clear task are available.
Key points
Section titled “Key points”- Jev returns decisions based on a state and a list of questions rather than generating text like a standard LLM; it is presented as zero-shot.
- Red Hat tested nine guardrails on prompt-injection detection and toxic-content detection.
- Qwen with 35 billion parameters led the prompt-injection test at 89.31%.
- Red Hat’s roughly 200-million-parameter DeBERTa classifier scored 89.01% on prompt injection and answered in about 54 milliseconds.
- Jev placed fourth on prompt injection with 86.35% and took around 348 milliseconds per call.
- Jev won the content-safety test with 86.2%, beating Red Hat’s small Granite Guardian model by about six points.
- Rewriting the prompt for the open-source LLaMA model improved its content-safety score by almost 18 points, while the same prompt made Jev perform worse.
- Red Hat’s conclusion was that decision models were not reliably faster, cheaper, or more accurate than using a conventional LLM as a judge.
- The video argues that a small model running on a CPU can be difficult to beat when labeled data and a clear task are available.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | What Jev is and what Red Hat tested | The video introduces Jev’s decision-based approach and Red Hat’s comparison of nine guardrails. |
| 01:01 | Prompt-injection results | Qwen led the test, while Red Hat’s small DeBERTa classifier nearly matched it and Jev placed fourth. |
| 01:32 | Content safety and prompt engineering | Jev won the content-safety test, while prompt changes helped LLaMA but hurt Jev. |
| 02:05 | Latency caveat and overall conclusion | The video notes Jev’s extra network latency and summarizes Red Hat’s conclusion about decision models. |
| 02:37 | Why small models still matter | The speaker argues that small, fast models remain compelling for clear tasks with labeled data. |
Quotes
Section titled “Quotes”one of the models that almost beat everything was a tiny 200 million parameter classifier running on a MacBook CPU. — 00:00
Jev came fourth at 86.35% and took around 348 milliseconds per call. — 01:01
the prompt that works for one model can quietly do worse on another one. — 01:32
decision models weren’t actually reliably faster, cheaper, or more accurate than just using a plain old LLM as the judge. — 02:05
a tiny model on a CPU is still really hard to beat. — 02:37
Concepts and entities
Section titled “Concepts and entities”Jev, TypeSafe, Red Hat, DeBERTa, Qwen3.6 35B A3B, NVIDIA NeMo Tron, Llama, Diffusion Gemma, Granite Guardian, LLM as a judge
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 So, everyone’s been hyping up Type-Safe’s new Jev decision model as the future of AI guardrails, so Red Hat decided to actually put it to the test. And honestly, the results are kind of hilarious because one of the models that almost beat everything was a tiny 200 million parameter classifier running on a MacBook CPU. So, here’s how the test went. So, idea behind Jev is pretty cool. Instead of generating text like a normal LLM, you give it a state and a list of questions and it just gives you back decisions. It’s also zero-shot, so you don’t need to train it or anything, and Type-Safe’s whole pitch was that you
00:30 can get an LLM as a judge with the same quality for a lot less money and a lot less latency. So, the Red Hat AI safety team lined up nine different guardrails and threw two problems at them: catching prompt injections and catching toxic content. In the lineup, you had some pre-trained classifiers, big LLM judges like Qwen 3.6 35 billion and Nvidia’s NeMo Tron, an old zero-shot model from 2019, and then the decision models, which were Jev plus two open-source alternatives, LLaMA and Diffusion Gemma. On prompt injection, Qwen with 35
01:01 billion parameters came in first at 89.31%, but Red Hat’s own little DeBERTa classifier, which only had around 200 million parameters and was literally running on a laptop CPU, got 89.01%. That’s like 0.3 points behind a model that’s over 100 times bigger. And it answered in about 54 milliseconds. Jev came fourth at 86.35% and took around 348 milliseconds per call. Now, to give Jev some credit, it actually won the content safety test at 86.2%
01:32 and it beat Red Hat’s own small Granite Guardian model by about six points. Red Hat was pretty upfront that they need a better small model for that one. But, the part that I found most interesting was the prompting. The open-source LLaMA model did really badly on content safety at first, around 58%, so the team rewrote the prompt specifically for LLaMA, and then it jumped almost 18 points. Then, they gave the same prompt to Jev, and Jev actually got worse. So, even with these fancy zero-shot decision models, you’re still stuck doing prompt engineering, and the prompt that works for one model can quietly do worse on
02:05 another one. But, to be fair to Jev again, it was being called from the UK to servers in the US, so every request picked up at least 56 milliseconds of extra network time. But, even accounting for all of that, Red Hat’s conclusion was that decision models weren’t actually reliably faster, cheaper, or more accurate than just using a plain old LLM as the judge. So, where does that leave us? Well, I think decision models make a lot of sense when there’s no ready-made classifier for the thing you’re trying to catch. But, if you’ve got labeled data and a clear problem, a tiny model on a CPU is still really hard
02:37 to beat. And personally, I’m just happy to see people getting excited about small, boring, fast models again, instead of throwing a giant LLM at every problem. And if you want to stay up-to-date with the latest in AI research and engineering, be sure to subscribe to the Better Stack channel.