Jeva - the Jev Competitor: This 4B Model Never Writes a Word. I Tested It

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Jeva - the Jev Competitor: This 4B Model Never Writes a Word. I Tested It · Channel: Prompt Engineer 48 · 06:24 · Watch on YouTube

Topics: Model inference and optimization, AI models, Local and open-source models

Summary

The video tests Jeva 4B, a model designed to select, classify, and score options without generating text. On the creator’s RunPod setup, it returned results in about 300 milliseconds after a slower first call and gave stable answers when option order was shuffled. The tests also found that Jeva correctly rejected off-topic requests but often chose “none” even when a suitable option was available. Reusing encoded options improved speed, while the creator cautions that their small tests do not verify the paper’s broader benchmark results.

Key points

Jeva 4B is built on a Qwen 3.54B backbone and supports null, choice, and rubric-based score questions.
The project was described as early, with code on GitHub under an Apache 2.0 license and no releases at the time of recording.
The creator ran the model on a RunPod A40 with 48 GB of memory; loading the checkpoint took about 20 seconds and peak memory was 10 GB.
After kernel compilation on the first call, a request with three questions took about 300 milliseconds and used zero output tokens.
Across 30 shuffled option orders, the choice did not flip; the largest probability change was about 7 × 10⁻¹¹.
Jeva correctly returned null for three off-topic requests, but it also returned null for real support requests that had matching teams.
In a small banking-intent test, disabling null allowed a duplicate-charge request to be matched correctly; enabling null led Jeva to select none.
Reusing options reduced the tested request latency from 102 milliseconds to 50 milliseconds, with the same answer.
Latency increased with larger candidate lists in the creator’s test, and the model chose the wrong intent for one request with 63 candidates.
The creator recommends checking null behavior on your own data or turning null off if the model must choose an available answer.

Chapters

Start	Chapter	Summary
00:01	What Jeva 4B does	The creator introduces Jeva’s option-selection and scoring approach, its design principles, and the hardware needed to run it.
02:12	First results and order stability	A support-ticket example shows Jeva’s outputs and latency, followed by a test of whether shuffling options changes its answers.
03:46	Testing the null option	The creator finds that Jeva rejects off-topic requests well but can also choose none when a valid support category is available.
04:49	Option reuse and scaling	The video measures the speed benefit of reusing encoded options and tests latency and accuracy with larger candidate lists.
05:53	Who should use Jeva	The creator summarizes where Jeva may be useful and advises checking its null behavior on your own data.

Quotes

it did not write a single word. — 00:01

output tokens, zero. — 02:43

So, this proves it is stable, not that it is right. — 03:16

it is very good at saying none and a bit too eager to say it. — 04:16

check the null behavior on your own data first, or turn null off. — 06:24

Concepts and entities

Jeva 4B, Jev 1.13, Qwen3.5 4B, RunPod, reutilização em cache, opção nula

Full transcript (original language)

00:01 This is one request to a 4 billion parameter model. It gave me three answers and an outage probability of 0.93, the team descended to, which is engineering, and an impact score of 1.98 out of two. All in about 300 milliseconds, and it did not write a single word. All right, in this video I test Geva 4B, and hold on to the none of the above number, because that one surprised me. So, this is Geva from Chi Lee and Sin Chuan at the National University of Singapore, paper 2610, 18773 on Archif.

00:37 The model page says it selects among alternatives, checks whether a statement is true, or scores something against a rubric without generating lengthy explanations. It sits on a Qwen 3.54B backbone. The code is on GitHub, about 75 stars when I recorded this, Apache 2.0 license, one contributor, and no releases yet, so treat it as early. The idea is four rules. The order of the options should not change the answer. You can add more options without a new output layer. None apply is a valid

01:09 answer. And options you have already processed should be reusable. Now, mind you, this is not going to run on my laptop. The install section says Linux and Nvidia GPU and Python 3.12 with CUDA 12.8 kernels, and I’m on Windows with an 8 GB GPU. A 4 billion parameter model in BF16 is already around 8 GB before anything else. So, I rented a pod on RunPod. I wanted the cheapest 24 GB card, the 4090 at 34 cents an hour, but there was

01:41 no stock, and the A40 with 48 GB at 59 cents an hour was available. So, I used that. I installed PyTorch 2.9.1, the requirements, and ran the doctor command, and you can see it says the kernel forward passed, and the issues list is empty. Loading the checkpoint took about 20 seconds and the peak memory I measured was 10 GB. Here is the example from the readme. You give it a state, which is the ticket text, and then typed questions.

02:12 The first is null, a yes or no. The second is choice, pick one of billing, engineering, or general. The third is score with a rubric from 0 to 2. The ticket says the production login page returns 503 and all customers are blocked. You can see outage probability 0.93, engineering at 0.64, with general at 0.34, which is a bit less sure than I expected, and impact 1.98. The very first call took 36 seconds

02:43 because the kernels compile, so mind you do a warm-up call. After that, the whole thing with all three questions took about 300 milliseconds. And look at the usage line output tokens, zero. It is reading the options and the state, not generating text. The paper claims zero flips under tested permutations. So, I tried it myself, same request, my payment was taken twice and three candidates. I shuffled the candidate order 30 times. Zero flips and the largest change in any probability was about 7 * 10 to the

03:16 minus 11, so that is floating point noise, mind you. The answer it kept giving was none, the same overeager null you will see in the next test. So, this proves it is stable, not that it is right. For comparison, the page shows another model, Jev 1.13, flipping on 17% of choice items. I only tested Jeva. So, that comparison is from the paper, not from me. Now, the number I told you to hold on to, Jeva can say none apply with a separate coverage estimate. I gave it

03:46 three teams, billing, engineering, and shipping, and six requests. Three are real support issues, three are not, like a chocolate cake recipe and who won the football match. All three off-topic ones came back null at .93, .93 and .99. That part is really good. But look at the first three. I was charged twice for my order has a billing team right there and it said none at .99. The parcel one, same thing .99. Only the app crash got an honest split, .52 null.

04:16 I thought maybe my wording was the problem, so I tried 16 banking intents. The style of the banking 77 said the paper uses. With null turned off, I got charged twice picks duplicate charge correctly. With null turned on, it says none at .92. Two of my four real requests got matched card arrival and change pin. So in my small test, it is very good at saying none and a bit too eager to say it. This is my own wording and about 10 requests, so it is not a benchmark. The paper reports 98.66% null detection AUC on banking 77 and I

04:49 did not reproduce that setup. The last property is reuse. I loaded four long support policies and asked three questions. The first one, the cold one, took 755 milliseconds and encoded 427 tokens of options. The same four options again, 50 milliseconds, zero tokens encoded. Then a new subset of three of them, 50 milliseconds with 319 tokens served from the cache and the same request with the cache off was 102 milliseconds. So that is 2.05 times faster and the

05:21 answer did not change, cancellation both times. The paper says about two times on type decisions and 27.98 times on its long document set at full reuse. I only reproduce the small one, so I can’t confirm the big one. I also checked how it scales with the number of options. Cache off, with five candidates it took 101 milliseconds. 16 candidates, 100. 32 candidates, 154. And 63 candidates, 300 milliseconds. At 63, it picked crypto purchase for a

05:53 card block request, which is wrong. So, my list of near duplicate intents is a hard test. Take that lightly. The paper tests up to 2048 options at 69.88% accuracy and 12 seconds median latency. And that is not something I ran. So, who is this for? If you route tickets, tools, or intents, and you want probabilities, not paragraphs with a stable answer under reordering, it works as advertised in my tests. If you needed to commit to an answer

06:24 when one exists, check the null behavior on your own data first, or turn null off. The weights were open when I downloaded them without a login, even though the project page says access controlled. If you’re looking for cheap and reliable GPUs, I use RunPod, and the link is in the description. I pulled the pod down as soon as I had the footage. Let me know what you want me to test next, and I’ll see you in another one.