Nokia Just Released a Free Jev Alternative (AnyJev)

Author: Better Stack · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Nokia Just Released a Free Jev Alternative (AnyJev) · Channel: Better Stack · 15:25 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Model inference and optimization, AI models, Local and open-source models

Summary

The video introduces AnyJev, a Nokia research team’s open-source Python library that adds decision-model behavior to open-weight language models without training or labeled data. In a benchmark of satire and real-news headlines, hosted Jev outperformed Qwen with AnyJev on accuracy, calibration, and speed, while AnyJev eliminated changes caused by rearranging answer options and runs locally for free. The bias correction helped on some headlines but could overshoot and hurt performance depending on the mix of satire and real news. The presenter sees AnyJev as useful for private, local decision tasks, but recommends testing it against real answers before production.

Key points

A decision model returns probabilities for typed options, rather than generating a text response that must be parsed.
AnyJev is a Python library from Nokia’s applied research team that adds a decision layer to open-weight models and supports several inference backends.
AnyJev estimates the probabilities of answer labels from next-token likelihoods; its L0 mode addresses position bias and label bias without labeled examples.
In the 500-headline benchmark, Jev achieved 80% accuracy, compared with 68.4% for raw Qwen and 68.6% for AnyJev L0 with the default batch prior.
AnyJev L0 had a zero flip rate when the answer options were reordered, while raw Qwen changed answers on 4% of the benchmark headlines.
Jev was better calibrated in the test: its calibration error was 0.07, compared with about 0.3 for Qwen and L0.
Using the content-free prior helped AnyJev remove Qwen’s strong satire bias on one sequential test, but it lowered the overall score on the 500-headline benchmark and hurt performance on a later test.
AnyJev runs locally and for free, which may be useful for private data, but the presenter advises validating it with real examples before production.

Chapters

Start	Chapter	Summary
00:00	Introducing AnyJev	The presenter compares models judging whether headlines are satire and introduces AnyJev as a way to add decision-model behavior to open models.
01:35	How decision models and AnyJev work	The video explains decision-model outputs, AnyJev’s supported backends, and its approach of scoring answer options from next-token probabilities.
03:08	Biases and the L0 correction	The presenter describes position and label bias, demonstrates how option order can change a model’s answer, and explains how L0 attempts to correct these effects.
05:12	Benchmark setup and results	The video introduces the 500-headline dataset, compares Jev with raw Qwen and AnyJev L0, and reports their accuracy, speed, cost, and flip-rate results.
07:44	Bias correction and calibration	The presenter examines Qwen’s tendency to predict satire, compares batch and content-free priors, and discusses how well the models’ confidence matches their accuracy.
09:47	Sequential headline tests	Further tests on selected headlines show that content-free correction can help remove a label lean but can also produce incorrect predictions.
12:50	Conclusions and practical advice	The presenter summarizes the trade-offs, notes AnyJev’s local privacy and cost advantages, and recommends checking its results before deployment.

Quotes

So, the model never generates text, and the answer can’t fail to parse. — 02:37

But L0 didn’t change a single answer. — 04:42

It stops the model from being fooled by the order of the options, and it removes the model’s lean towards one label — 13:22

Just make sure you check it against some real answers before you deploy it to production. — 14:23

Concepts and entities

AnyJev, Jev, Qwen3, L0, content-free prior, llama.cpp, Nokia, RoBERTa

Full transcript (original language)

00:00 Okay, before we start, I want you to read this headline and decide for yourself whether it’s a real news article or a joke article from The Onion. Take a second because these are harder than they look. So, this one is real and there are three models here that try to guess whether it’s satire or not and all of them tell a completely different story. A regular Qwen model running on my MacBook says there’s a 99% chance it’s satire. Jev, which is a hosted decision model, honestly doesn’t know and says that there’s a 58% chance

00:31 it’s a real news article. And the exact same Qwen model with a new library called AnyJev on top of it says that it’s 100% real news. Now, try this one. This one is also real, but this time it fooled every single model including Jev at 86% satire. To be fair, it fooled me, too. Now, what is this AnyJev that we’re looking at here? Well, AnyJev is a new library developed by Nokia’s applied research team. Yep, the same Nokia that made the famous 3310 and it claims you

01:03 can turn almost any open model into a decision model like Jev with no training and no extra labeled data. So, I ran a data set of 500 headlines from The Onion and from real news sites through all of them to see if they can confidently identify what is satire and what is not. And the results were a lot more interesting than a simple winner or loser. So, in this video, we’ll take a look at AnyJev, see how it works, and then we’ll do a detailed comparison of all these three models and see how they

01:35 perform. It’s going to be a lot of fun, so let’s dive into it. So, here’s a quick reminder of what a decision model is. A normal LLM writes you an answer word by word, but a decision model just makes a quick judgment call. You give it some state, which is the thing you want it to judge, and a typed question, which can be a choice, a yes or no, or a score. And you get back a probability for every option, so there’s no text to parse. Jev does

02:06 this through a hosted API. It costs 4.2 cents per million input tokens, and the output is free. And in my test, each call was about 310 tokens, so 1,000 calls cost a little over a cent. But the problem with Jev is that it’s closed source, so you can’t download it or run it yourself. AnyJev is a Python library that adds this decision layer on top of any open-weight model running on your own machine. It works with Hugging Face Transformers, vLLM, SG Lang, and

02:37 llama.cpp. And it’s licensed under the Apache 2.0 license. And the way it works is actually pretty simple. It shows the model your state and your question, with every option given a letter, like A for satire or B for real news, and then instead of letting the model write anything, it looks at how likely the very next token is to be each of those letters, and that becomes your probability. So, the model never generates text, and the answer can’t fail to parse. And here’s how you set it

03:08 up on your own machine. First, you pick a back-end, and then you wrap that back-end into a decider, and this is where you choose the level, so raw for plain output or L0 for the bias fix. We’ll talk about that in a second. And after that, you define your question with its options, and you call decide with your text. What comes back is a probability for every option, plus the level it was answered at. So, what is that L0 I just mentioned? Well, when you read the raw probabilities for a model,

03:38 they have two biases built in. The first one is position bias, which means the model can prefer an answer just because of where it sits in the list. The second one is label bias, which means some answer labels just feel more likely to the model, no matter what you’re asking. Here’s what position bias looks like. This is the exact same headline and the exact same question, and the only thing I changed is the order of the two options. And this headline is a real news article. When satire is listed first, RoBERTa gives satire only 1%, so

04:12 it’s sure the headline is real. Then, I swap the two options and now it says 99.8% satire, so it completely changes its mind. But any Jevs L0 says 100% real news both ways. And Jev only moves from 25 to 23% satire. And across all 500 headlines, RoBERTa changed its answer on 20 of them when I swapped the options, which is 4%. Jev changed on 18, but those were tiny moves of about three

04:42 percentage points on average. So they were mostly close calls to begin with. But L0 didn’t change a single answer. So L0 fixes this bias issue in two steps. First, it asks the question once for every ordering of the options and averages the results, so no option gets an advantage from where it sits. Then, it estimates how much the model leans towards each label and divides that lean out. And it does all of this without a single labeled example. And there are

05:12 two ways it can estimate that lean. The default one is called batch prior, which looks at the answers across a batch inputs. And the other one, called content-free prior, which measures the lean from an input with no real content in it, which is what you would use if you only had one headline to judge. Keep that second one in mind because it’s going to matter a lot in the test. So for the test itself, I’m using a data set from Kaggle called news headlines data set for sarcasm detection. It has

05:42 28,619 headlines where the satire comes from The Onion and the real ones come from Huffington Post. And the labels are really clean and trustworthy because The Onion only publishes satire, so nobody had to guess which headlines are jokes. And I took 500 headlines, half satire and half real, and every model gets the same question. Is this headline satire from The Onion or is it real news? And we have three contestants here on screen, Jev 1.13, which runs through

06:12 Type Safe’s API, and the other two are running on my M2 Max MacBook Pro through Llama.cpp. And they’re using Qwen-3, the 8 billion parameter model with the raw readout, and I’m using the exact same Qwen model with AnyJev’s L0 on top of it. And I built this little app so you can watch it all happen live. So let’s start the run. So, as you can see here, Jev is going through about 10 headlines per second, and it finishes all 500 in 51 seconds, and that’s including the

06:43 network latency from my Mac to Type Safe’s servers. At that point, my laptop is still only about a third of the way through. per second, and AnyJev L0 does under three because it reads every headline twice, so it’s slower. And both models are sharing the same GPU here. So the whole run on my laptop took about 4 minutes and 36 seconds. And as you can see, the final numbers are 80% accuracy for Jev, 68.4%

07:14 accuracy for Raw Qwen, and 68.6% for L0. So on accuracy, Jev still wins by about 12 points, and AnyJev’s fix barely changes the score. And here’s the full scoreboard for the benchmark run. Jev wins on accuracy, calibration, and speed, and the whole run cost me less than a cent. But AnyJev wins on flip rate with zero, and on cost because it’s free. And there’s a fourth row at the bottom, which I’ll come back to in a

07:44 second. So, when I looked closer at the mistakes, I found that Quinn leans really hard towards satire. It called almost 69% of the headlines satire when the real number is exactly 50. So, it gets 87% of the satire right, but on the real news, it’s basically a 50/50 split. And this is where the fourth row comes in. This one is L0 with the content free prior, the one I mentioned earlier would matter in the tests. So, with the default batch prior, L0 barely changes

08:15 Quinn’s answers. Because Quinn is almost always at zero or 100%, and correcting something that extreme for a lean rarely flips it. But the content free prior corrects a lot harder. It gets about 67% of the satire right and 66% of the real news. So, the lean is completely gone. But it gave up more satire answers than it won back on the real news. So, overall, it ends up two points lower at 66.6%. But the part that I find most

08:47 interesting in this chart is when I grouped every answer by how confident the model was, and then I checked how often it was actually right. If a model is well-calibrated, then when it says 80%, it should be right about 80% of the time, and the line should follow this diagonal. So, Jeff follows the diagonal really nicely. Quinn and L0 end up almost on top of each other, and nearly all of their answers sit in two big dots at the very ends. In fact, 87% of raw

09:17 Quinn’s answers came with 99% confidence or more, and it was wrong on 30% of those. So, its confidence tells you almost nothing about when you can trust it. And L0 doesn’t change that, which is actually expected because the docs describe L0 as removing biases, not as calibrating confidence. Its calibration error is 0.3, almost the same as raw, while Jeb’s is 0.07. Now, let’s make the test a little bit harder. In this round,

09:47 I have seven real articles that sound like The Onion, including the High Rocks and the coffee ones that we saw at the beginning, plus three actual Onion headlines mixed in. And since I’m judging these headlines one at a time, there’s no batch to measure Quen’s lean from, so any Jeb needs to use the content free prior in this round. So, let’s look at this one. Amazon customer stunned after their about you profiles claim they have no friends or flat buttocks. So, this one is a real

10:17 headline, and Quen says it’s 96% satire. Jeb sits right on the fence at 52 to 48, and any Jeb says it’s 100% real, which is correct. Let’s try this headline. 80% of the people watching this video are not subscribed to the Better Stack channel. Well, look at that. That is interesting. So, if you see yourself as one of those people in that bucket, then I would consider subscribing so you don’t miss out on other cool videos like this one. And here’s one of the three Onion headlines I mixed in. Rockstar

10:48 announces GTA 6 soundtrack contains 10,000 songs by Randy Newman, and Jeb is 100% sure that this one’s satire, and raw Quen agrees, but any Jeb now says 99.5% that it’s real news. So, at the end of this round, any Jeb wins by 70%. Jeb gets 50%, and raw Quen gets 40%, and that’s exactly what the L0 promises to fix. So, raw Quen leans heavily towards satire, and it called nine of these 10

11:18 headlines satire, but L0 removes that lean. So, on real headlines that sound fake, it fixed four answers that the raw Quen model got wrong. It only broke one, which was the GTA headline. Now, there’s one fair criticism on this test. The data set is from 2019 and it’s been all over the internet, so there’s a good chance that these models saw some of those exact headlines during training. So, for this round, I picked 10 recent headlines, half of them from The Onion and half real ones. So, let’s see how

11:49 that goes. Corn named world’s sexiest plant. This is an easy start. Every model identifies it correctly as satire. HBO announces return of hit show a decade after its final. This one is real and every model gets that one, too. But, then when we get to late uncle remembered as man who had can crusher in garage, which is satire, Jev says 75% satire and Raw Kwenn says 100%, but Any Jev now says it’s 76% real news. And my

12:19 favorite one is this one. Authorities investigate collision between trains from Chicago traveling 80 mph, DC traveling 75 mph. Which is basically a school math problem written as a news story and Raw Kwenn gets the joke at 98% satire. Jev barely gets it right at 52% and Any Jev says it’s 99.8% real news. So, in this round, Jev gets 90%, Raw Kwenn gets 80% and Any Jev

12:50 drops to 60%. So, that’s the same correction from round one working against it this time. So, with an even mix of satire and real news, removing Kwenn’s satire lean cost it two satire headlines that Raw Kwenn got right and it didn’t fix a single one in return. So, what did we learn from all of this? Well, on the big test, Jev is about 12 points more accurate. It’s faster even when you account for network latency, but that depends on your machine and

13:22 it’s confidence actually means something. So, for less than a cent per 500 headlines, that’s a pretty good deal. But, I would say any Jev does exactly what it says. It stops the model from being fooled by the order of the options, and it removes the model’s lean towards one label, which made it win in one of these sequential rounds. But, those sequential rounds also showed that there is a catch to this method because removing a lean can overshoot it in the other direction. And whether that helps

13:53 you depends on the mix of answers in your data. It also doesn’t make the model smarter, and the default L0 mode doesn’t calibrate the confidence. But, any Jev does have higher levels that do calibrate this, but those need a few hundred labeled examples, and I didn’t test them here. I guess the big advantage is that everything ran on my laptop for free with no data leaving my machine. So, if your data is private, any Jev on top of a strong open model is probably a good library to use. Just

14:23 make sure you check it against some real answers before you deploy it to production. Because as we saw today, it does get it wrong sometimes. There’s also a little app that I published on GitHub where you can try this on your own. If you set it up, you can type any headline into the input box, and it will show you if it’s real or satire, and you can find the link to that project in the description below. So, there you have it, folks. That is any Jev in a nutshell. I think the idea behind it is really cool because now you have a library that you can use to turn any

14:54 open weight model into your own decision model, but I was kind of disappointed by the test results. On the Qwen 8 billion parameter model, the training free approach wasn’t enough to catch up with Jev. So, I’m not sold on it yet. But, those are just my thoughts on the topic. I would like to hear yours, so let me know what you think in the comment section down below. And folks, if you like these types of technical breakdowns, please let me know by smashing that like button underneath the video, and also don’t forget to subscribe to our channel. This has been

15:25 Andrus from Better Stack, and I will see you in the next videos. >> [music]