This New AI Architecture Makes Decisions 13x Faster

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: This New AI Architecture Makes Decisions 13x Faster · Channel: Prompt Engineering · 16:22 · Watch on YouTube

Topics: AI agents, Model inference and optimization, AI models

Summary

The video introduces contrastive language models (CLMs), an architecture designed to make decisions by matching a situation to a set of candidate actions rather than generating an answer token by token. The presenter explains how CLMs are trained, why cached candidate embeddings can make decisions much faster, and how the model performs in demos involving support tickets, tools, and Wikipedia links. Experiments show that latency stays low as the number of options grows, but accuracy can fall when many similar candidates must be distinguished. The presenter suggests using CLMs to shortlist options before a more deliberative model makes the final decision.

Key points

A CLM uses separate encoders for the current state and possible actions, then scores actions by their similarity to the state.
Yes/no, multiple-choice, and scoring queries can all be represented as a state paired with a closed list of candidates.
Contrastive training brings correct situation-action pairs closer together and pushes other actions farther away; examples in a batch can provide negative candidates.
The model is trained in three stages: general question-answer data, hard negatives, and agent trajectories with some original data mixed back in.
The 8-billion-parameter backbone remains frozen; small trainable heads adapt its embeddings, allowing the dataset to be embedded once.
Because candidate actions can be embedded and cached, subsequent decisions can be very fast even with many options.
The presenter reports that latency changed little as the candidate list grew, while accuracy dropped substantially when choosing among 1,080 tools.
Since each option is evaluated individually rather than compared side by side, similar candidates can confuse the model.
A suggested use is to let a CLM quickly reduce a large candidate set to a shortlist, then use a model that reads the options together to make the final choice.

Chapters

Start	Chapter	Summary
00:00	Why fast decision models are attracting attention	The video introduces system-one models and presents the Stanford and Nvidia contrastive language model as a different approach to fast decisions.
03:22	Contrastive training and the source of speed	The presenter explains positive and negative training examples, cached action embeddings, and the latency advantage as candidate lists grow.
06:34	Accuracy, training stages, and model design	The video discusses the limits of judging options independently and describes the CLM’s training stages and frozen backbone.
09:48	Demos on a DGX Spark	The presenter demonstrates CLM on support-ticket classification, a large tool list, and a Wikipedia link-racing task.
13:06	Trade-offs and practical use	The experiments show accuracy falling with large candidate sets, leading to a proposal to use CLM for shortlisting before a slower model decides.
15:15	Multimodal plans and a broader comeback	The presenter looks ahead to a multimodal model and reflects on the renewed interest in fast models and contrastive methods.

Quotes

The easiest mental model is CLIP, but for decisions. — 01:39

Each new step costs one embedding for the state plus a dot product for options — 05:30

At almost a thousand twenty-four options, it’s forty-four milliseconds against five hundred and seventy milliseconds for Jeff. — 06:03

Each option is encoded on its own, so it never gets to compare them side by side. — 13:36

let CLM cut thousand tools or candidates down to top 10 in milliseconds — 14:43

Concepts and entities

CLM, Jev, CLIP, Stanford, NVIDIA, DGX Spark, Qwen3, vLLM, Gemini, RTX 4090, aprendizado contrastivo, hard negatives, System 1 e System 2

Full transcript (original language)

00:00 With the release of Jeff, everybody now is talking about system one models. Non-thinking, non-reasoning models are suddenly hot again. And the idea makes a lot of sense. Most of the decisions inside real softwares don’t need a model to think for 30 seconds. Things like which tool should run next? Is this ticket urgent? Uh which link should the agent click? You want those answers in milliseconds rather than seconds or minutes. We have already seen quite a few open-source replications of Jeff or ones

00:33 that claim to be. But most of them are still language models under the hood. It reads your text, reads every single option, and then writes out an answer. Even if the answer is just a single word with some probability associated with it. Now, this week a new one landed with a very different architecture. It’s called a contrastive language model or CLM. This is from a team at Stanford and Nvidia research. And it’s extremely different than what we have seen so far. They’re making some bold claims, saying

01:09 up to nine times faster than Jeff or on par in accuracy. But I’m not making this video to do a head-to-head with Jeff. What makes this model interesting is how it’s built and the scale it’s built at. It’s a 8 billion parameter model, the same size as a lot of those clones. Now, the interesting part is that the speed does not come from making the model smaller. It actually comes from the way it is designed and architected. So, in this video, not only I want to explain

01:39 how this is different, how it’s built, and what’s interesting about it, but I’ll also show you a working demo of this running on a DGX Spark. Okay, so what exactly is a CLM? The easiest mental model is CLIP, but for decisions. If you remember CLIP, it puts images and text into the same embedding space. For example, a photo of a dog lands right next to the word a photo of a dog. CLM does the same thing with situation and action. There’s a state encoder which reads what’s going on right now

02:15 and an action encoder which reads what you could do. Take Mario with a Goomba walking towards him. The state is the game screen and the actions are left, jump, or run right. To make a decision, you embed the state, embed every option, and score each option by how close it lands to that state. Then, you run a softmax over those scores and you get a probability for every action. For example, things like jump gets 80%, right might get 15%,

02:49 left might get 5%, and in that case, Mario is going to jump. They have designed it such that it speaks the same Jevs API. So, you get three different primitives, a yes/no question, a multi-choice question, and a score. Now, all of these things are exactly the same underneath, a state plus a closed list of candidates. Yes/no is just two candidates, true and false. Okay, so what this means for you as a builder? Now, just like Jev, it can never invent an answer that is not on the list. But,

03:22 it can still pick the wrong answer from that list. And when it does, it can do that confidently. That is the same line which Type-Safe used that Jev can never hallucinate. It’s true for CLM as well. But, it doesn’t mean it’s always right. So, how do you get a space where the right action actually lands closest? The solution is to train it contrastively. Training starts with a lot of examples. Each one is a pair, a situation, and an

03:55 action somebody actually took. Uh let’s take the example of the Mario game. So, Mario sees Goomba, and the player has to jump. Now, the model job is simple. Put each uh situation close to its own action and far away from every other action. Now, if you do that millions of times, in the situation ends up sitting right next to the moves that fit them. That’s how you train the model. Now, here’s the nice trick. You don’t need

04:27 anyone to write down wrong answers. See, you grab 1,000 examples at once. For Mario screen, jump is the right answer. And the other 999 actions in that file belong to other situations. So, they’re automatically wrong for this one. And this is essential for contrastive learning. You want both positive and negative examples, and in this case, you are getting those negative examples for absolutely free. Okay, so here’s where the speed comes

04:59 from. With a Jeff style model, uh the state and all the options go into the model together on every single call. Now, if you double the options or make them longer, you’re going to end up need to read twice as much. You are going to need to decode the answer token by token out of a uh large language model, which gets really expensive really quickly. On their chart, constrained decoding with 238 billion climbs to over 4 seconds. CLM

05:30 splits them apart. In most agent loops, the actions don’t change from step to step. The state changes. So, the idea is that you embed the actions once and cache them. Each new step costs one embedding for the state plus a dot product for options and a dot product in this case is essentially free computation. In their Mario demo, uh that takes each step from five forward passes down to one. Now, you can see it in this chart. As

06:03 the number of candidates goes from one up to a thousand, CLM’s line barely moves. At almost a thousand twenty-four options, it’s forty-four milliseconds against five hundred and seventy milliseconds for Jeff. So, we’re looking at about thirteen times faster. Now, in my system one versus system two video, I said system one scales in width. Uh lots of small questions answered at once. Now, CLM pushes that about as far as it

06:34 will go. With a thousand options for roughly the price of one. But, there’s a catch. And it’s the classic trade-off with this kind of two encoder design. Because the state is encoded on its own, it never sees the options. It cannot lie them side by side and compare them. Each option is judged on its own individually. Now, this is a big problem. We’re going to talk about it in a minute. But, you might be thinking, we already have embedding models, so why not just

07:07 use those? Now, here’s why. If you ask plain Aquaint 38 embeddings who wrote Romeo and Juliet, and William Shakespeare actually comes in in third, which is supposed to be the right answer. So, similar is not the same as right. After CLM’s training, Shakespeare comes first. And it all comes down to how this model is trained. It’s trained in three different stages. Stage one is sixty million question answer pairs from Nvidia’s NeMo tron data.

07:39 Uh the question is the state and the answer is the action. That’s where it learns general knowledge about the world. Stage two is where it gets interesting. They used Gemini to generate 30 million hard negatives. These are the answers that are close, but wrong. Things like Venus instead of Mercury for the planet closest to the sun. Now, this is important because this is what teaches the model the difference between sounds right and is right. And the order really matters. On a test full

08:13 of hard negatives, pre-training alone gets 52%. Now, when they add the hard negative stage on top, it actually jumps to 69%. But, if you train on hard negatives from day one, it simply peaks at 62% and then starts overfitting. So, you learn the world first and then the fine distinctions second. This is like pre-training and then post-training. Now, stage three turns it into an agent. That’s about a million steps of real

08:44 agent trajectories, where each step is um something like here’s my context and here is what I did. And then mix 40% of the original QA data back in. Without that replay, the hard negative scores fall from uh 69 to 56. Now, here’s the part that actually kind of surprised me. The 8-billion model is frozen. It’s just a queen 3 HP used as a reader, and it now gets a single gradient. The only thing that trains is

09:15 a small head on each side, which are about 20 million parameters. Because the backbone never changes, you embed the whole data set once, and after that, a full pre-training run takes about an hour on a single RTX 4090, which is kind of incredible. And it does uh scale predictably. The loss follows a power law in compute and data, in head size, and in encoder size. Encoder size gives the biggest gains of the four. So, the obvious next step is a bigger backbone,

09:48 and they say a multimodal CLM is coming early next month. Okay, that’s enough theory. Now, let me show you how it works on a single DGX Spark. Okay, let me show you some of my demos. So, all of this is running on a DGX Spark. It’s running the Quinn 3.8 billion encoder served through VLLM. On top of that sits the CLM server that the team actually released. And now, the server itself is about 75 MB, but the whole thing takes about 25 GB of memory. I give it a

10:21 state, in this case, a support ticket from a customer who got charged twice and can’t reach anyone. Then I asked three typed question in one call. Is this urgent? Which team should handle it? And how frustrated is the customer? And you can see every answer comes back as a probability. Urgent has 84% probability. Billing has 99%. Frustration is very angry, right? And once it’s seen this ticket, asking again comes back

10:53 under 2 milliseconds because everything is cached. So, this is one of the beauties, like if you provide it with this list of options or set of tools, everything is going to be cached and the subsequent responses going to be extremely fast. Now, remember the Romeo and Juliet example that we mentioned in the beginning? We have same encoder, same candidates, and the only difference is that uh that small head on top, right? So, with raw Quinn embeddings, uh Christopher Marlowe comes first and Shakespeare second. Now, if we switch to CLM, Shakespeare’s probability jumps to

11:28 98%. So, those uh 20 million parameters of training really do reshape the space from similar to right. Now, with this demo, the architecture actually really shows up. So, I built a list of 1,080 tools, things like Stripe, Twilio, uh QuickBooks, and so on. So, for contacts, Jev caps a choice to 255 options. So, this won’t even fit into a single call there, right? Now, the first request takes a few seconds because CLM

12:00 embeds every tool once. But, after that, every new request takes about 80 milliseconds. And here’s the part that I love. Picking from one tool or uh from 1,000 uh it’s going to basically cost you the same amount of time when it’s once it’s cached. Now, this one is a fun one. Wiki racing code is not released, so I rebuilt it myself. You start on one Wiki page, and you have to reach a target by clicking links. So, here it’s plate tectonics to bioluminescence.

12:36 On every page, CLM scores every single link towards the target. We are In this case, we’re talking about 934 links on this first page, all scored in one go. Then it picks biosphere, then biome, then plant con on the plant con page bioluminescence comes up first at 58% right? Basically, took about four clicks. In their own demo, I think they reported around six clicks. So, this is

13:06 actually pretty impressive. So, here’s what I took away from my experiments with CLM. It actually scales beautifully with compute. The number of options barely changes the latency because the options are embedded once and then cached. Now, what does not scale the same way is accuracy. As the number of options grow, so with eight tools to choose from, it picked the right ones 86% of the time in my experiments. But, with all 1,080 tools,

13:36 that actually dropped to 17%. Mostly because it confused tools that sounded alike, like several different refund tools. Now, the same thing showed up in the Ubi Key racing. So, some races it nails, others it wanders off. And that’s the catch we talked about earlier. Each option is encoded on its own, so it never gets to compare them side by side. Also, some of these examples that I tried were a little extreme. You do not want to be putting 1,000 tools for a agent to select from. You probably want

14:11 to create sub-agents where you have a different set of tools available to each sub-agent. So, overall, I think it’s a very interesting direction for system one models. They have turned the decision into retrieval, which basically encodes the situation, then looks up the action. And that makes a thousand options cost about the same as one in terms of latency, at least. Okay. So, as a builder, the way I would use this today is as a first stage. So,

14:43 the idea would be let CNN cut thousand tools or candidates down to top 10 in milliseconds, then let a model that reads the option together make the final call. So, here, the fast thinking or system one models are used as short lists, and then careful decisions are going to be followed by system two. And if your set of actions is already small and stable, like a game loop or a handful of routes, it actually works pretty well. Now, this

15:15 version is text-only, but there’s a 35 billion parameter model coming up, which is supposed to be multimodal. Now, I’m personally interested in that because their scaling curve says a bigger encoder is where the gains are, and a bigger encoder is exactly what should help it tell similar options apart. So, it’s going to be very interesting to see how that shapes up. But, towards the end, I actually want to share this thought. I feel like older algorithms are making a comeback. System one models

15:49 basically are fast, non-reasoning models which have made them hot again. And with this, they’re actually bringing back contrastive learning machines or contrastive algorithms, which were really hot in 2017-2018. So, I think they are also making a comeback. I’m actually pretty excited about it because I don’t think we have actually pushed the limits of existing algorithms. Anyways, do let me know your thoughts, how you think about system one and system two models, and what type of applications that you see in your own

16:22 workflows. Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one.