Context Language Models: Your Agent Doesn't Need Compaction

Author: Prompt Engineering · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Context Language Models: Your Agent Doesn’t Need Compaction · Channel: Prompt Engineering · 13:32 · Watch on YouTube

Topics: AI agents, Prompt and context engineering, Model inference and optimization, AI safety

Summary

The video introduces the Context Language Model (CLM), an approach that lets an agent edit its own context instead of relying on fixed-point summaries that may discard important details or hallucinate. A demo and experiments with Pi show how the model can maintain notes and update them as it reads files, with stronger results than Pi’s default summarization on the creator’s synthetic tasks. The video also explains practical setup requirements and limitations, including parallel tool calls, latency and token costs, model capability, and prompt-injection risks. The creator views CLM as a promising direction, provided the harness gives the model room to manage context and users monitor caching.

Key points

Conventional agent harnesses summarize conversation history near a context limit, potentially losing information that becomes important later.
CLM replaces the usual append-to-context process with a model-controlled function that can keep, shrink, delete, or rewrite context.
The Pi extension lets the model edit a context file, and users can inspect changes and context size in a panel.
The paper reports a 59.4% score on BrowseComp Plus for its approach, with about a fifth less compute than the best summary approach, as stated in the video.
In the creator’s synthetic warehouse-log task, Pi’s summaries introduced fabricated seal codes, while CLM tracked the codes accurately.
The creator notes that Pi’s safety net handled most context cleanup in initial tests; one tool call per turn and size notes after each result enabled the model to edit context.
Editing earlier parts of context can reduce prefix-cache reuse, increasing latency and generation cost; CLM was not automatically cheaper in the creator’s runs.
CLM’s effectiveness depends on model capability, and the video says the paper’s results were closer to a tie with summaries at a 128,000-token budget.
Because the model can write to its own memory, prompt injections may persist across turns.

Chapters

Start	Chapter	Summary
00:00	The context bottleneck and a CLM demo	The video frames summarization as a risk to important details, then demonstrates Pi replacing a long log in its context with a concise note.
01:33	Why fixed-point summaries can fail	The creator explains how summaries can lose details that matter later and introduces CLM as a model-managed alternative.
03:42	How Context Language Models work	The model edits a context file between turns, and the video describes strategies and benchmark results reported in the paper.
05:45	Running CLM and testing warehouse logs	The creator explains the Pi extension and test setup, then compares default summarization with CLM on synthetic log tasks.
09:24	Results and practical limitations	The video reviews CLM’s task results and discusses harness settings, cache reuse, cost, model capability, and prompt injection.
13:01	Conclusion	The creator reflects on CLM’s promise and invites viewers to share their context-management approaches.

Quotes

Context management is one of the real bottlenecks for agents that we have today. — 00:00

The model decides what the entire context is going to look like on the next turn. — 03:42

So, the agent has no way to know it’s wrong. — 08:23

And one more thing on security, if the model can write to its own memory, so can a prompt injection. — 12:28

Concepts and entities

Context Language Model, Context Bench, Pi, Meta, University of Washington, BrowseComp Plus, vLLM, SGLang, suffix cache reuse, prefix caching

Full transcript (original language)

00:00 Context management is one of the real bottlenecks for agents that we have today. As they get close to the end of the context window, we force the agent to summarize everything that has happened so far. And the summary can throw away the most relevant information. So, what if instead of summarization, we let the agent edit his own context window? That’s exactly what the team at Meta and University of Washington did in a new paper called the context language model. Now, before I show you how it works, let

00:31 me show you a quick demo. So, here’s Pi, the open-source coding agent. It’s running locally on my DGX box. So, I have given it a stack of warehouse shift logs to go through, which is a lot more than that can fit in its context window. Now, normally at this point, the harness would step in and write a summary in order to preserve the context. But look at what is happening here. The agent opens a file. That file is its own context. And it takes this 3,000 token

01:02 log it read a few turns ago and replaces it with one line, which preserves the exact information that context is providing. And if you look at this panel, you can see the context size climbing at every file it reads and then dropping every time the model cleans up after itself. Now, the beauty is there are no rules what to keep and what not to keep. The model is deciding what exactly is the most important information. Okay, so in the rest of the video, I

01:33 want to talk about how exactly does this work and how you can run it yourself. And then we’re going to talk about some gotchas I ran into while testing it because there is one cost that’s really easy to miss. All right, but let’s first talk about why this matters. Every agent harness today, like Cloud Code, Codex, or Pi, handles a full context in a very similar way. At a fixed point, say 75% of the window, the model writes a summary and the high earnest keeps those last few turns and

02:06 basically replaces the previous conversations with that summary. If you use Codex or Cloud Code, you can actually trigger this with the {slash} compact. But, here’s the problem. Say your agent is debugging a service and say 40 turns ago it read a configuration file uh with one value that turns out to matter the most later on. Now, the summary was written before anyone knew it’s going to matter. If that value didn’t make it into the summary, it’s basically gone and the

02:37 agent doesn’t even know what is missing. And the agent might spend a lot more time in order to dig through the logs or well, spend a lot of tokens trying to figure that out again. Now, the paper built a small benchmark around this called Context Bench, which includes simple tasks like keeping a Sudoku board up to date while the moves streams in. Now, on this simple benchmark, most of the existing compaction techniques actually fail. Now, one way to tackle this is you could

03:09 trigger smarter summary prompts or compact more often. Now, this could potentially help, but the problem is that the compaction still happens at a specific point and the summary is still written. At the end, you’re going to be end with a summary of summaries, which is going to potentially lose all of the most important information that you want to preserve. So, what exactly is a context language model? In a normal language model, every turn adds to the end of the context.

03:42 So, essentially the new context that is fed into the model on the next turn is the old context plus whatever the model produced. Now, a context language model replaces that plus sign with a function. The model decides what the entire context is going to look like on the next turn. So, it can essentially keep, shrink, delete, or rewrite anything it needs. Okay, so here’s how it actually works. So, before every request, the harness

04:14 writes the context out to a file. Then every turn is a block with a header. Now, the model can open that file, edit it with a small Python script exactly the same way it would edit code. And whatever is left in that file at the end of the turn becomes the next prompt. That’s exactly what we saw in the demo. Now, when you hand a model this kind of control, it comes up with its own strategies. In the paper, it invented a

04:45 new chat role called notes just for itself. It wrote a little helper function to compact old search results and called it 37 times. And in a multi-agent run, it kept a scoreboard of 21 sub-agents inside its own context and updated it in place 163 times. Now, the beauty is that you can actually trigger compaction uh with very simple instructions in plain English. For

05:15 example, if you say something like, “Once you pass 24,000 tokens, compact it down to 4,000.” And it actually worked. Now, this is enough to actually change exactly when the model is going to compact. And according to paper, this works on BrowseComp Plus, which is a Deep Research benchmark, it scored 59.4% with 20.627B. That’s about 11% higher than and best

05:45 summary approach with about a fifth less compute. Okay, so how do you run this yourself? The authors actually released an extension for Pi and you can install it with a single command. Then inside Pi, you can type {slash} CLM. This is going to open up a panel you saw in the demo with the context size over time and a page that shows every edit the model made side by side. Okay, so for my experimentation, I ran it with quant 3.8 27B, which is served

06:19 locally with VLLM. Now, though the paper actually used a previous iteration of the 27B, so I’m using the latest version. They had a 32,000 tokens token budget for the model, so we’re going to stick to the same budget as well. I’m going to put the exact settings in the description if you are interested in replicating this. Now, my goal was to see how this actually holds up in real tasks. So, I built three different tasks that didn’t really

06:49 fit in the window and ran them against Pi’s own summarization. Okay, we’re going to look at those, but here is one example. I described the warehouse logs in the beginning of the video. These are basically 16 days logs and the agent has to track truck seal codes. Then it actually needs to figure out which one got voided a day later or after a few days and list the ones that are still valid. Now, this is a synthetic example. Don’t treat it as a

07:20 benchmark, but I actually found some really interesting things the context language model does with this. Okay, so to establish a baseline, we used the Pi summarization and there were some really interesting things that came out of the baseline itself. Pi summarizes three times and then we We every summary against the logs. Okay, so the first summary listed 12 seal codes, but the problem was that 11 of those uh didn’t exist anywhere. So, it

07:51 basically hallucinated those codes. Now, by the third summary, 14 of 18 codes were basically made up. It even invented a little sequence, which was right next to the real code. Now, this is one of the main problem with summarization of summaries, because you are going to lose a lot of information, and the agents, especially the smaller models, can hallucinate a lot. Now, the summary actually looks perfectly confident with the check boxes

08:23 and the next steps. So, the agent has no way to know it’s wrong. Now, the interesting thing was that the agent actually got the answer right uh despite those summaries, but the reason was not that it remembered things. It actually went back and really reread every file from disk with grep, even though the prompt told it not to do so explicitly. On another task after a summary, the agent decided that the logs didn’t contain any

08:54 deploy information, but in reality, the deploy line was right there. And it had already read it, but during the summarization process, it had completely ignored it. Now, this was with the default summarization in Pi. Now, let’s look at what happened with Pi CLM on the same tasks. The model wrote its whole context into one notes block. In this case, it keeps track of every seal code that was applied, every seal

09:24 that was voided, and what to read next. All 18 codes were exact. And across the seal and incident tasks, it got every answer right except one. While Pi needed the grip rescue or it basically hallucinated. Okay, so the best part is that it actually works. You can use it. However, you need to keep a few important gotchas in mind because if you’re going to be building on this, it’s important to know the

09:55 limitations than the strengths. And this is what I always suggest. Look at where the system failed, especially with these new language models. Okay, so the first one is that the hardness matters more than you would think. And three of the four default turns, the model never edited its own context at all. Pi CLM has a safety net that swaps old tool outputs for a short note when the context is about to overflow. And in

10:26 most cases, that safety net did all the work. The reason is parallel tool calls. The model read four files in one turn and then blew past the remainders before it could actually react. So, if you want to use this, you need to turn on one tool call per turn and the size note after every result, which is how the actual paper harness works. So, with those two settings, the model edited its own context. The second one

10:57 is cost and latency. So, when a server like VLLM reads your prompt, it saves its work in a cache. And if nothing can actually reuse that prompt prefix caching. Now, for context language model, it really matters where the model is changing its context file. So, say if it’s changing context towards the end of the file, it can still preserve and reuse most of the context

11:27 as a prefix caching. The responses are going to be faster. However, say if it starts editing the file in the middle, then anything beyond that point is going to be lost in the prefix caching, which means it has to reprocess everything beyond that point. It will not only increase the latency, but also the cost of generation. And the third one is that it’s not automatically cheaper. So, in one of my seal runs, Pi CLM processed about twice as many token as

11:58 Pi. And on another another one, only 10% of the prompt came from the cache. Now, the paper also built a fix called suffix cache reuse that keeps the work for unchanged text after an edit. But for now, uh it’s an SGL lang patch for a one model on one GPU. So, I couldn’t really test that. And the fourth one is that the model has to be good at this. In the paper, uh the 9 billion parameter

12:28 model was six points behind clean summaries until they trained it with reinforcement learning. And their headline results uses a small 32,000 token budget. At a 128,000 tokens, CLM and the summary basically is a tie. And one more thing on security, if the model can write to its own memory, so can a prompt injection. And any prompt injection is going to stay across turns. This is one of the

13:01 concerns that the paper itself highlights. So, overall, I really like this direction. Summaries are written once and then trusted forever. And in my runs, uh that’s exactly where things kind of went wrong. Letting the model manage its own context fixed those problems for me. But the main thing is that as long as the harness gives it the room to react, and uh you keep an eye on the cache. It’s a very interesting approach to be honest, right? We’re going to see I think there are going to be evolutions

13:32 of this, but let me know in the comments what you would use this for or if you have seen anything like this or have come up with approaches that you’re using for managing the contacts window, please do share them in the comments section below. Anyways, I hope you found this video useful. Thanks for watching and as always, see you in the next one.