Build RAG with JEV

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Build RAG with JEV · Channel: Prompt Engineering · 11:05 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, RAG, search and information retrieval

Summary

The video demonstrates how Jev can support decision-making in retrieval-augmented generation (RAG), focusing on reranking retrieved documents according to changeable policies or other instructions. It explains Jev’s decision primitives and shows how criteria can distinguish passages that directly answer a question from those that merely share its topic. The presenter compares Jev with cross-encoder and LLM rerankers, highlighting instruction-following, concurrency, and cost considerations. In the demonstrated benchmark, adding Jev as a reranker raises top-one accuracy over a BM25 baseline from 21% to 54%.

Key points

The video focuses on using Jev at decision points in a RAG pipeline, especially for reranking.
Jev offers binary decision-making, choice ranking, and a score ladder.
Criteria can encode policy-specific decision boundaries, allowing rankings to change as organizational policies change.
The example query asks how to fix a 401 unauthorized API error, with multiple candidate chunks and metadata.
Jev can distinguish a direct answer from passages that only overlap in topic or keywords.
The presenter reports that concurrent decisions using 64 pools took 7.6 seconds, compared with about 40 seconds when run sequentially.
Cross-encoders are described as unable to follow additional instructions, while LLM rerankers can follow them but may be slower and more expensive.
In the presented test, Jev reranking improved top-one accuracy from 21% with BM25 full-text search to 54%.
The presenter suggests Jev can be used as a drop-in replacement at decision points in existing workflows.

Chapters

Start	Chapter	Summary
00:00	Why use Jev in RAG?	The presenter introduces RAG as a practical setting for Jev and outlines its decision-making primitives, with a focus on reranking.
02:39	Setting up the retrieval example	The video describes a document retrieval system where changing policies can make semantically similar results incorrect under current business logic.
05:18	Using criteria as executable policy	The presenter demonstrates how criteria can classify direct answers separately from passages that only share keywords or topics.
07:57	Comparing rerankers and costs	The video compares Jev’s steerable ranking with cross-encoders and LLM rerankers, discussing instruction-following, speed, and cost.
10:03	Retrieval accuracy and takeaways	The presenter reports the BM25 and Jev reranking accuracy results and closes by describing Jev as a potential workflow decision-point replacement.

Quotes

jev turns out to be a perfect candidate for those decision points. — 00:32

the criteria provides strict decision boundaries. — 05:18

you can actually steer your reanker with this. — 08:27

if you add Jev as a reranker, it pushes it up to 54%. — 10:03

Concepts and entities

Jev, RAG, reranking, cross-encoder, BM25, Gemini

Full transcript (original language)

00:00 Jev, it seems to be the only thing people are talking about right now, but I wanted to create a video showing you practical examples of where exactly you can use this because there is not much content on it to be honest. So I decided to combine it with one of my favorite topics which is retrieval augmented generation. Now it’s extremely boring but if you look at different steps in a true augmented generation pipeline you see that there are multiple places where you

00:32 have to make a decision such as reancher filter or citation and jev turns out to be a perfect candidate for those decision points. So in this video we’re going to look at how you can combine Jev with the retrieval of augmented generation for making these decisions and in the process I want to show you how to actually think about combining it this into your own workflows beyond the demos that you have been seeing on the internet. Now Jeff basically comes with

01:05 three different primitives that you can use. One is binary decision making. You can create two or false classifier. The second one is choice ranker. So you can have multiple different choices, define a criteria and based on that select a specific choice and then you have a list or basically a score ladder we where you assign different scores to different options. Now for this video we’re going to be mainly focusing on the reranking

01:37 which I think is a very interesting example which actually shows you different capabilities of Jev. One very interesting topic that I have been looking at is what if the policy of an organization changes but the actual context or the business logic remains the same. This is where most of the cross encoders actually fail. Now you can use something like an LLM reanker but it’s prohibitively expensive and I

02:09 found that Jev actually gives you a really great combination of sterility and extreme low cost. Now there are other examples like if you are building something like graph rag where you need to extract knowledge from a corpus you can actually use jev to validate entities that are extracted. I’ll probably cover this in another video. Now if you’re new to Jev and want to understand what exactly it does beyond

02:39 the hype I highly recommend to watch this video. Okay. So link to the notebook is going to be in the video description. Now the example we’re going to be trying to solve here is that there’s a two documented generation system as opposed to extract chunks or documents but there could be multiple different competing policies different versions and the system needs to resolve around them. When you run this block it’s going to ask you for the API key.

03:09 It’s dirt cheap. You actually get $5 of free credit when you sign up. Here I’m using the Python official SDK. You can also use the rest API if you like that. Then we have a number of different helper functions. You have query the retrieve documents and then the LLM generates a response based on those documents. However, in the middle there could be some policy changes, updates to documentation which may not be actually reflected

03:39 in how the answers are being generated because your vector distances can only measure semantic proximity but not correctness under your business logic. In a couple of examples I as I mentioned there might be changes to the official API docs or you might be looking for in-depth architectural explanation versus quick copy paste snippets. Okay. So in the rest of the video the benchmark that we’re going to be using. So there is um one query with multiple different candidate points and then

04:12 there are two different official uh versions of the policy which are going to change over time and we’re going to test multiple different options. One is the cross encoder then LLM reanker and then Jev with the null criteria and in the process you’re going to actually see the effectiveness of a classifier like Jev which has the ability to follow instructions. So for this example, our query is going to be how do I fix 401 unauthorized error from the APIs. And to

04:46 make it simple, I have a bunch of different chunks that I’m assuming are going to be returned from the either vector search or full text search along with a bunch of metadata. So this is what you would expect from a simple retrieval augmented generation pipeline. Now we want to create a reranker which is going to basically rank them and not only based on the user query that is provided but also additional instructions okay but first we need to understand primitives specifically how

05:18 do we use criteria as a executable policy now as I said there are three different primitives we’re going to be using null which basically assigns calibrated probabilities and there are two potential outputs whether it’s true or false criteria. Now in this case we’re going to be using criteria as a policy which instead of prompt in instructions the criteria provides strict decision boundaries. So here is a set of instruction which says as the passage directly answer the question.

05:50 Now there are two different criterias. So one is the specific answer the question asks for and false is share words or topics but does not answer it. So it’s not just true false. You can actually find the specific criteria you want. And just to show you how it works. So here I have a dictionary with multiple uh different keys. So for example, a direct answer would look something like this. Refunds are issued to the original payment method within 14 days. Right? And then there could be

06:22 another example which overlaps with the uh topic but doesn’t really answer it. There might be another one where we have keyword based matches, right? And then something which is completely off and unrelated. Now if we run all of those four queries through our null criteria, Jude actually assigns the highest probability to do the direct answer. And for the other the ones, you can actually see that the confidence is really low. Right? So this can be used as a classifier. you can define a threshold

06:54 uh on which you accept the answer below which you discard it. Okay. So before showing you my experiments I actually ran some ideas to see whether we can run multiple concurrent questions through this system and it seems like it’s actually great at concurrency. So for example, if you were to run multiple questions serially, in this case we are looking at 17 different decisions per second, you would get about 40 seconds if you run

07:26 those sequentially. However, uh if you just use their 64 pools that it offers, you can get all of the decisions within 7.6 seconds, which is incredible. Okay, in the rest of the video we’re going to use Juul as a reanker and I want to directly do a comparison with cross encoder then LLM as a reanker and jewel and see how effective it can be. Now in this case we are specifically providing it a criteria based on which it needs to

07:57 rank them and there are two different criterias. So the first one is let’s say a policy called authoritative version two and we’re going to be using the null primitive which means there are two different options and then there’s a second one which is fast workaround which is not as strict as the authoritative one. Now the main takeaway is that based on the criteria that you define the ranking of the documents that you’re going to get out of Juul is going

08:27 to be very different. So the criteria or the in steerable instructions that you provide here can really impact what type of ranking you get from the documents and this is the beauty because you can actually steer your reanker with this. Now traditionally we use cross encoders for reanking where you provide pair of question and the document or chunk right but it doesn’t really have the ability to follow any instructions. So it’s kind

08:58 of frozen in time. Okay. But you might be thinking there’s this second option where you can use LLM as a reanker which can actually follow instructions. You can tell it exactly what the criteria should be for ranking and that’s actually true. However, this is going to cost you a lot and it’s usually a lot slower. Now to show you an example, I ran the same prompt and query through Jev, Gemini flashlight and Gemini flesh. The cost is extremely high compared and

09:31 the most interesting thing is that the cost actually depends on how big the chunk size is. So for relatively small chunk size, both the uh feed and cost of Jev are very close to flashlight. However, as you increase the uh size, you start seeing cost pretty much remain like linearly incre. However, the biggest difference is when you start increasing the chunk size. This is where it actually shines. Okay. Now, one more thing um the

10:03 accuracy of the final response is really dependent on the accuracy of retrieval. So, in this case, I created a baseline which uses BM5 full text search. Um a set of queries that you can actually see the baseline accuracy is only 21%. But if you add Jev as a reranker, it pushes it up to 54%. And this is the top one accuracy we’re talking about. Right? So usually you are going to have uh say 10 or 20 different chunks and you compute

10:34 top K in which case this is going to be a lot more higher. Okay. So this was a quick example of how you can use a model like Jeff uh to replace different decision points in your existing workflows. Rag is a really good example of that and you can directly plug in Jeff as a drop-in replacement in your one workflows especially if you have to make decision at specific points. They don’t have to be Je’s binary decision.

11:05 Um I’m thinking of creating more content on show you you how you can use Jev as a a part of your existing workflows. If there is any specific topic you’re interested in, do let me know. I’ll be happy to cover that in the upcoming videos. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.