I Built My Own Jev for $3 (vs the Real One)

Author: Prompt Engineer 48 · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: I Built My Own Jev for $3 (vs the Real One) · Channel: Prompt Engineer 48 · 13:58 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models

Summary

The video demonstrates building a small, fast classifier inspired by Type Safe’s Jev, using Qwen 3.5 4B, Unsloth, and a rented GPU. Fine-tuning on roughly 38,000 examples cost about $3 and improved the model’s overall benchmark accuracy from 63% to 70.8%; it also outperformed Jev on the held-out test set. Jev still scored higher overall, had better-calibrated probabilities, and performed better on phishing questions, where fine-tuning made the home-built model worse. The video also compares latency and costs, noting that running the model near an application can be fast, while the hosted service can be cheaper at lower usage levels.

Key points

Jev is presented as a fast classifier that takes state, a question, and options, then returns a choice and probabilities.
The project uses Qwen 3.5 4B and a pipeline of about 38,000 questions drawn from several datasets and task types.
The model is fine-tuned with Unsloth and LoRA, with training labels applied only to the answer letter.
The A40 GPU rental cost about $0.49 per hour; the full setup, training, and testing took about six hours and cost around $3.
Fine-tuning raised the reported overall benchmark score from 63% for the base model to 70.8%.
On the held-out test set, the custom model scored 85.8%, compared with Jev’s 80.6%; Jev scored higher overall at 75.5% versus 70.8%.
The custom model reached 88.9% on ticket routing and 88.3% on tool-risk cases, while Jev scored 100% on the 27 ticket-routing examples.
Fine-tuning reduced the custom model’s phishing score from 60% to 47.7%, illustrating the importance of including relevant examples in training data.
Jev’s calibration error was lower (0.11 versus 0.21), while latency depended on where requests were run: from a laptop, Jev was faster in the reported test.
The video estimates that a continuously running GPU becomes cheaper than Jev at around 20,000 decisions per hour, while Jev is cheaper below that level or users can consider serverless.

Chapters

Start	Chapter	Summary
00:00	What Jev does and what the project will build	The video introduces Jev as a fast classifier and outlines a plan to fine-tune a small open model to return choices with probabilities.
02:06	Dataset and GPU setup	The creator describes the training data and tools, rents an A40 GPU, and prepares the project environments and dataset.
04:47	Baseline evaluation and training method	The untrained model’s benchmark results motivate a LoRA fine-tuning setup that learns to select answer letters and derives option probabilities from model scores.
07:28	Training, results, and publishing the model	After a smoke test, the creator trains on the full dataset, reports improved results, and uploads the merged model and adapter to Hugging Face.
09:06	Serving the model and comparing answers	The model is served through a vLLM API and compared with Jev on examples before the video runs a larger benchmark.
10:41	Benchmark, latency, cost, and verdict	The video compares accuracy, calibration, latency, and cost, then concludes that the custom model is useful for tailored use cases while Jev leads overall.

Quotes

the whole thing was trained on a rented GPU for about $3 — 00:00

Speed is not the problem. Accuracy is what we have to fix. — 05:18

Ours is 105 milliseconds and Jeff is 232 milliseconds. — 09:39

a small model is only as good as the data you give. — 11:46

overall Jeff is still ahead. — 13:27

Concepts and entities

Jev, TypeSafe, Qwen3.5 4B, Unsloth, RunPod, NVIDIA A40, LoRA, vLLM, Hugging Face, Together

Full transcript (original language)

00:00 All right, in this video, we will build our own Jev. So, you can see here I asked a plain Qwen 4 billion model, “A customer got charged twice. What does the customer want?” And it says, “Cancel subscription, 59.6%.” That’s wrong. Now, after fine-tuning the same question, “Duplicate charge, 99.8%.” And mind you, the whole thing was trained on a rented GPU for about $3, and then we’re going to compare it against the real Jev from Type Safe uh both in intelligence and latency. So,

00:31 let’s go ahead and do this. So, first, what is Jev? If you have seen my earlier videos, you already know this. But for those who are new, Jev is from Type Safe, and they call it a system one model. System one model means fast thinking. It doesn’t write you a poem. It does not write long paragraphs. You give it a piece of state, some text, and a question, and it gives you back a type answer. That can be a yes or no, a score, or a pick from a list. And you can see on the blog, they say it run uh it runs in about 70 to 500 milliseconds,

01:03 and the price is about 4.2 cents per million input tokens. And the output tokens are free. That’s really cheap. And if you look at the API here, it’s very simple. You send a state, a question with criteria, billing, technical, sales, and it returns the choice plus the probability for each option. That probability part is really important because your code can decide what to do when the model is not sure. Now, why would you want your own? Mind you that Jev is a hosted model. You call their API. If you want to run it on your

01:33 own GPU, keep your data private, or just understand how these things are made, you need your own version. And this is exactly what we are going to do today. So, here’s the plan. We take a small open model, Qwen 3.5 4 billion, and we teach it to behave like Jev. You give it state, a question, and options, and it answers you just one letter, A, B, C, plus a probability for each option. For the training data, we need lots of questions of this type. There is an open data set pipeline for exactly this, and I will put the link in the description.

02:06 It pulls around 38,000 questions from well-known data sets, multi NLI, which is support, contradict, or neutral. BullQ, which is a yes or no. Banking 77, which is picking a banking intent. Then we have news classification, sentiment, and there’s a lot of policy and writing questions. And every question is converted into the same format, which is question, state, options, and one letter as the answer. And for training, I’m going to use Unsloth, which is free and open source, on a RunPod GPU. You can check the link in the description. Then

02:37 we push the model to Hugging Face, so that you can go ahead and use this as well. And finally, we put it head-to-head against the real Jeff. So, step one, we need a GPU. I’m going to use RunPod. I pick an Nvidia A40, which has 48 GB of VRAM, and it costs about $0.49 an hour. You can use 49D as well. Anything with 24 GB or more works. Now, mind you, your own laptop, or my own laptop, has an 8 GB 4060. A 4 billion model can technically fit on a in 4-bit

03:09 format on that, but it’s really tight, and it’d take me about 6 to 8 hours. For this video, we go ahead with the cloud. So, we are inside the pod now. You can see Nvidia SMI says A40, 46,000 MB, and there are all the files in our project. I will leave the link to all these files in the description. And there’s one setting file, the .env file. Here is the template, your Hugging Face token, the repo name where the model will be uploaded, and the type safe key for the comparison. Never share your real keys. I’m only showing the empty template

03:39 here. Step two, I’ve made one script, setup_pod.sh, and that does everything for you. Let me show you what’s inside. It installs UVI, which is a fast Python package manager. Then it creates two separate environments, one for training with Unsloth and for serving another for serving with vLLM. Now, why two? Because they’re these libraries want different versions of PyTorch, and if you put them together, they will fight with each other. So, keep them separate, it saves you a lot of headache. And then the third part, it clones the dev1 repo from Together and builds the dataset. It

04:13 downloads the source dataset from Hugging Face and converts each question into the same format. So, I’m going to go ahead and run this. This does need some time, a few minutes. All right, done. You can see 37,842 training questions and 4,568 test questions. Let’s look at one of them. So, here the state is a paragraph about zero-hour contracts. The question is, do I have to work on zero-hour contract? And the options are A for no, B for yes. And the correct answer is A. That’s it. That’s one training example. The model will see 37,000 of these. Now, before we train anything, I wanted to see how good the

04:47 plain model is. So, I served the untrained 3.5 4 billion with vLLM and asked for our questions. First one, the customer says, you charged my card twice for October uh then we say 19.99 both on the same day. And the options are duplicate charge, cancel subscription, card decline, or none. And you can see the base model says cancel subscription 59.6% duplicate charge, which is the right answer, gets only half a percent. That’s completely wrong. Then an easy

05:18 one, returns are allowed within 30 days. The purchase was 12 days ago. Is it within the window? Yes, 99.4% so easy questions are fine, tricky and 10 questions are not. And I also ran the full benchmark on the base model 747 questions overall 63% on the held-out test set 63.9 fishing email 60, ticket routing 70.4, and tool risk 70. It’s already uh and it’s already fast about 113 milliseconds per answer. So, speed is not the problem. Accuracy is what we

05:51 have to fix. So, let’s look at the training script, and don’t worry, I’ll explain every part in simple words. First, we’ll load the models with Unsloth model or fast model {dot} from pre-trained Gwen 3.54 billion. Unsloth makes the training around two times faster and uses less memory. Then, we add low-rank adaptation or LoRA. What is LoRA? Instead of changing all 4 billion numbers inside the model, we attach a small adapter layer and those. It’s like adding sticky notes to a book instead of rewriting the book. Rank eight, alpha 16, small and light, which keeps training cheap. Then, the

06:24 data. And here’s the important trick. Each example has a prompt and a completion where the completion is just the letter. We set the labels, so the models only learn from the answer letter. The question part is masked with minus 100, so the model is not trying to memorize the questions, and it only learns to pick the right letter. Then, before training, we test the base model. After training, uh we test again. So, we always have a before and after number. And then, the training settings. Learning rate is 5e-5, one epoch, batch size is eight, cosine schedule. And here’s the part. How do we get

06:57 probabilities like Jeff? When the model is about to answer, it has a score for every possible token. We just look at the scores for the letters A, B, C, D and turn them into percentage with softmax. So, we get the full probability for every option for free in one single step. No need to generate any text. And at the end, we save the model, merge the LoRA into it, and push it to Hugging Face with one line, push to hub merge. Now, before spending hours, always do a smoke test. I trained it on only 200 examples just to make sure that nothing

07:28 crashes. And honestly, this saved me because the first time it crashed on a setting that the new version of Transformer removed. So, fix, run again. And you can see uh it works end to end. It loads, tests, trains, tests again, and merges the model. With only 200 examples, score doesn’t move much. It actually went a little bit down, which is normal. We need the full data. So, now the real deal, all 37,840 examples, one epoch. I’m going to go ahead and start it. This will definitely take some time. On the A40, it took

08:01 about 4 hours 40 minutes for 7,300 steps. In the meantime, if you look at the loss here, it starts showing 0.35 and goes down to around 0.13. And the orange line, the loss on question is never trained on, goes down to about 0.1. Lower loss means the model is getting better at picking the right letter. The whole pod, including setup, training, and all the test was up for about 6 hours, so about $3 in total. And And everything is fully open on Sloth on your own GPU. You own every file. And when training finishes, it tests again

08:34 itself. Before, uh 72.6% after 87.2%. That’s a really big jump for one epoch. And look at the tricky ones. The policy question went from 53 to 95% and the routing questions from 55 to 88. And then it uploads to Hugging Face automatically. So, here is the model on Hugging Face, prompt48 /myjeff4b. You can see the model card with the after and before results, the per data scores, and the copy-paste code again. Then the full merged model from VLLM and the separate LoRA adapter version if you

09:06 want to train further. So, anyone can go ahead and use this right now. Now, let’s serve it. I’m using VLLM, which gives you an OpenAI compatible API. So, any code that talks to OpenAI can talk to our model as well. One command, bash serve.sh, it downloads the model from Hugging Face and loads it on the GPU. And we say and we wait until it says healthy, and then we are ready. And on RunPod, if we expose port 8000, you get a public URL, and you can go ahead and call it from your laptop or your app. Now, the moment of truth. Same questions

09:39 that the base model got wrong, the double charge, and our model says duplicate charge 99.8%. That’s really good. Earlier, it was half a percent. Now let’s ask our model and type save Jeff side side by side. For the double charge, Jeff also says duplicate charge 100% and our model says 99.8. Look at the time. Ours is 105 milliseconds and Jeff is 232 milliseconds. And mind you, ours is running on the same machine here as so no internet in between. I’ll show you the fair compensation comparison in a moment. Sentiment. The headphones

10:10 sound fantastic and fit perfectly. I love them. Ours says very positive 95% with a little bit of positive. Jeff says very positive 100%. Yes no question. The museum is closed on Monday. Is it open on Monday? Ours says no 99.9% and Jeff says 100% no. So on the simple examples, both get it right, but a few examples prove nothing. We need the big test. So I went ahead and made a benchmark script with 747 questions, 360 from the held out test set which the model never saw

10:41 it training. Then three benchmark made by other people for Jeff, 300 fishing emails, 60 tool risk cases. Then we have 27 support tickets. Our model was never trained on those. So it’s a fair out of distribution test. Now for every question, we record is it correct, the probabilities, and the time it took. All right, here are the results. Overall, the base model is 63% our model is 70.8% and Jeff is 75.5%. So we improved a lot, but Jeff is still ahead. But this is our model. And now look at look at it one by

11:14 one. On the held out test set, ours is 85.8 and Jeff is 80.6. So on the type of questions we trained on, we actually beat Jeff. Tool risk, ours is 88.3 and Jeff is 91.7. Really close. Ticket routing, ours is 88.9 and Jeff gets all 27 right and it’s 100% correct. Fishing, this is interesting one. Our base model was 60% and our model went down to 47.7%. Jeff is 64, so fine-tuning made it worse on fishing. Why? We never trained on any security or email

11:46 questions, so the model got really good at our sort of questions and a bit worse at what was the new type. So, it’s really important to know a small model is only as good as the data you give. If you want fishing detection and fishing examples, uh you have to add fishing examples to the training data. Now, calibration. This measures when the model says 90% sure, it’s right 90% age of the time. Lower is better. Our calibration error is 0.21 and Jeff is 0.11. So, Jeff’s probabilities are more honest. Mind you, I trained Jeff with a special method just for this and we did

12:20 just fine-tuning, so this is expected. And latency, mind you, this benchmark ran on the pod, so our model is on localhost and Jeff goes over the internet. That’s not fair for Jeff, so I ran it again from a laptop where both goes over the internet. On the pod, ours was 118 milliseconds and Jeff was 269, but from my laptop, ours was 461 and Jeff was 357. So, over the internet, Jeff is actually faster. Most of the time, it’s the run pod proxy in between. The model itself takes around 100 milliseconds, so if you run it close to your app on the same server, it’s really fast. And cost, Jeff is 4.2 cents per

12:54 million input token, which came to about 2.3 cents per thousand decisions. Ours on a 49 cent and our GPU came to around 0.4 cents per but if the GPU sits idle, you’ll still pay 49 cents every hour. So, the simple rule, if you make more than around 20,000 decisions an hour on a 24/7 pod, your own model is cheaper. Below that, Jeff is cheaper or you can go serverless. So, what’s the verdict? For about $3, we took Gwen from 63% to 70% and on our own type of questions it

13:27 even beat Jeff, but overall Jeff is still ahead. It’s better calibrated and handles brand new questions types like fishing much better. Jeff is trained on a special method for calibrated probabilities and it’s a production service. So it don’t manage any GPU. Our model is open. It’s yours. You can run it offline and train it on your own data. If you train it on your company’s question, you definitely get better results on your use case. So in this video we have seen how we can build our own Jeff, a fast system one classifier

13:58 with Unsloth on a rented GPU, push it to Hugging Face and compare it with the real Jeff. The model and all the scripts are linked in the description. Go ahead and try this out. Let me know how it goes. Data set and tool link are in the description. I will see you in this next one.