The Boring Version of Continual Learning

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: The Boring Version of Continual Learning · Channel: LangChain · 20:35 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, Software engineering with AI, AI models

Summary

Will Brown argues that continual learning for models will become a routine part of software development rather than a sci-fi process. Agent experiments need to mature into production workflows supported by metrics, evaluations, monitoring, and feedback from users. Teams can use methods such as distillation and reinforcement learning to improve models for specific tasks, while weighing capability against cost, latency, and performance. As post-training practices become better understood and automated, model development will increasingly follow familiar software life cycles.

Key points

Continual learning is presented as a practical extension of software development, not an inherently futuristic process.
Experiments with agents can reveal useful workflows, but production systems need versioning, releases, rollbacks, and attention to cost, latency, and performance.
User behavior and feedback help teams identify new capabilities, failure modes, and regression tests.
Evaluations should capture the human vibe check and provide scalable measures for comparing models and improving workflows.
Distillation and reinforcement learning offer different ways to adapt models, each with trade-offs; neither is a universal solution.
Making systems evaluatable by default and collecting production traces can support maintenance and model improvement.
Specialized models may be preferable to general-purpose models when teams need better fit, cost, or control for a particular workflow.
Post-training is currently difficult, but its processes may become routine and automated as best practices emerge.

Chapters

Start	Chapter	Summary
00:03	Continual learning as the next normal	Brown frames continual learning as an important future practice that will become as ordinary as building software.
03:13	From agent experiments to production	He explains how informal agent experimentation becomes a mature production workflow with code, releases, evaluation, and monitoring.
06:23	Feedback, metrics, and evaluations	User behavior and qualitative checks can help define evaluations that measure model performance and reveal useful improvements.
09:35	Distillation and reinforcement learning	Brown discusses distillation and RL as model-improvement methods, highlighting their trade-offs and potential uses.
12:44	Building evaluatable systems and choosing models	He advocates making evaluation native to production systems and choosing whether to buy, fork, or specialize based on the use case.
16:25	Model development as a life cycle	He describes a cycle of specifying objectives, training, testing, staging, releasing, and monitoring models, with specialization offering potential savings.
19:01	Automating post-training	Brown compares today’s difficult post-training work to manual software processes that can become routine and abstracted through automation.

Quotes

continual learning is boring — 00:03

it starts looking less like a tool and more like software. — 04:17

The goal should be that the eval captures the vibe check. — 06:56

RL kind of is the button to do this. — 11:11

the boring version of it is just that continual learning is software development life cycles for models — 20:35

Concepts and entities

Continual learning, Will Brown, Prime Intellect, evals, SFT, On-policy distillation, Offline context distillation, RL, Claude Code, Codex, Cursor, GitHub Actions

Full transcript (original language)

00:03 I’m Will Brown. I lead applied research at Prime Intellect. And my goal today is to convince you that continual learning is boring. And by this, I mean like you should do it. It’s where things are going. It’s the future. But it’s also like it’s becoming less sci-fi very quickly. And I think with what I’m going to talk about today, my goal is to kind of paint a picture of it that is going to feel more mundane. It’s going to feel regular. It’s going to feel like just, oh, of course, is the goal.

00:34 This isn’t where we are today, but this is kind of the picture of where I think we will be not too long from now in terms of how continual learning for models fits into the products and deployments of companies like the ones I’m sure many of you here are building or working at today. And it’ll be borrowing lessons from software. So software now is boring. Software is a commodity. Software is free. Software is dirt cheap. The cost of software is going to zero. And yet we all build software. And soon I think models are going to look a lot like that. And so the exciting version of learning, the version that I think a lot of us like

01:06 when we read blogs and think about new algorithms are like imagining in our heads is like, oh, we have an agent deployed in production and it’s designing its own training recipes. It’s like optimizing its own kernels. Weights are going to update every single token that flows through, and like it’s going to be like doing multi-tenant hot swapping of all these different adapters for memory modules and combining all these fancy recipes and figuring out like what to do on the fly with no human in the loop, and it’s like literally AGI. And I’m not saying no to this. I’m saying yes to this, but I’m saying this is also the other side of a coin that looks much more boring. And this is just kind of, I think, in many ways the natural conclusion of like the

01:38 path we’re on today. And like as the models we’re using get better and better at the rate that they are, like this is the sort of thing that like is going to feel just pretty normal not that long from now. And I just, I don’t think it’s going to feel like this explosive world-changing thing, except in the way that it already does. Like it already does feel wild that the models are doing what they’re doing. But iPhones were like that not so long ago. The cloud was like this. Cars were like this. Like, we have technology that is incredible and changing the

02:10 world all the time in many ways. And we get used to it really quickly. And so, that’s kind of what I mean by this claim, is that we’re going to get used to it. It’s going to be normal. It’s going to be fine. It’s going to be amazing. And I want to think about tools versus software as like the way that we interact with our tools today and the way we interact with agents today and the kinds of companies people are building. A lot of us, our primary experience with agents is in the terminal, or it’s with some product that is kind of a nice glossy GUI on top of a terminal experience, be

02:40 it Cursor, Claude Code, or Codex, or things like this. And we play with agents. We’re having a lot of fun. We’re like orchestrating them by like hopping on all these tabs and like having a bunch of terminal sessions open, and we’re like writing our prompts, and we’re saving our skill files. And it’s like really cheap to play. We get to experiment quickly. We get to have a lot of fun rewriting our skills, and like we see the models do a thing and then we’re like, “Oh, this is great.” And we keep pushing it to the extreme and we realize it fails and we’re like, “Oh, the models aren’t quite there yet.” And then we walk back to a thing because we got overconfident, and then we complain about quantization, even if it’s just like getting used to the models being

03:13 good and then pushing them beyond their limits. And we don’t really eval it beyond this sort of vibe check. We’re not really measuring what we’re doing with our agents day-to-day most of the time. Like I don’t know too many software developers who could tell me their pass rate of their agents that they’re using for coding. It’s just how often do you feel like you have to yell at the system you’re building, and how often you have to change your skill files and AGENTS.md and these sorts of things. But then in production, when we’re building companies around agents, well, a lot of the ideas we’re doing for these agent products come from this experimentation where in the terminal we’re playing with things. We’re trying stuff out and we see, oh, if I chain together these tools and have a

03:46 workflow that looks like this, I actually want to run this all day. I want to run this not just like on a one-off thing, but I’m going to have, it’s going to be a scheduled task, or it’s going to be a recurring workflow, or it’s gonna, I’m going to bake it into a skill and have a cron job that like runs this every morning. But these are also what the products that we’re building look like. They’re, the products are agents in production that are taking ideas that we experimented with from seeing what the models are capable of and what the emerging capabilities of the last training run from the labs has been. And we’re turning this into a thing that can then be packaged more easily and given to more people to give them a new experience around using AI.

04:17 And we’re orchestrating this via code. We’re calling the agents via APIs. We’re doing versioning. We’re doing releases and rollbacks. And we’re taking the developer approach of authoring specs more systematically. We start thinking about eval. We start thinking about using prompt optimization. And we start caring about cost, latency, and performance. And if you’re shipping an agent to prod that is serving tons of users, you want to notice regression. You can’t do the vibe check at scale for all your users all day. You don’t want to just wait for people to complain. You need to have a process. And this is how it, what it looks like for an agent product or an agent workflow to become mature, is that it starts looking less like a tool and more like software.

04:50 And this process is how we discover the workflows. Like we are doing these experimentations to understand like what are the agentic programs that are possible today, which is changing very quickly. And so we’re always iterating on this. We’re always trying a new thing. We’re always rewriting our software. But like rewriting software is not a new phenomenon. Like look at any repo that you use, any library, and look at like the history of the versions of it over time. Like it’s changed a lot. Like there’s a lot of things that happen because one thing changes that means another thing has to change. And this domino effect keeps, this domino effect just cascades throughout all of

05:20 software that’s interconnected. And so a lot of it is maintenance. It’s making the thing, you want to add features. You want to make it better at a thing, but you also have to keep up with whatever, everything else that’s moving around it. And this is what happens in production when you ship software. And this is also what we’re realizing has to happen in production when you ship agents. As I was mentioning, users will see a thing, they’ll love it, they will find that it does one thing, they’ll get excited and try a harder thing that it actually can’t do yet, and then they’ll complain, and then this gets fed back into a thing that you want the thing to be able to do. It’s a new feature idea. It’s a new capability.

05:51 It’s a new regression test. And so this experimentation loop is both how we build our agents, but also once we ship an agent product to users, it’s what they’re going to do, and it’s going to be what’s driving this, the distribution shift, the cycle of improvement of where the feature set should be going for the products we want to build. This is just like, this is not an AI-specific thing. This is like when you give someone a new website, a new phone product, people explore, people try stuff out, people figure out what they think makes sense, what they would like it to be able to do. They get new ideas from seeing what it can do, and they want it to be different in a

06:23 new direction, and so you have to keep up. And so defining the goal is just from getting this stream of feedback that comes from the metrics you collect about user behavior and how long people stay on the page and like how fast is your reload time and like how often are people retrying the same thing. And you take all these, and as we’ve kind of all talked about already tonight, we want to turn these into evals. Evals are kind of like the metric that allow us to kind of look at, is the model doing model stuff correctly for what I care about? And the evals are great because you can use them for everything. You can turn them into harness optimization loops for choosing your skills.

06:56 You can use it to just pick which model. Do you want to use Claude or GPT for this workflow? You probably want to eval. And just like pick whichever one does best on your eval if your eval is good, which means you need to make sure your eval is good by doing some higher-level grounding in like doing the vibe check based on the eval. And the goal should be that the eval captures the vibe check. Like the vibe check on its own is not a scalable thing, but it’s a very useful iterative process to uncover what the eval should be. Because like we just, at some point you have to go look at what the model says and check it, and this also starts telling you what is the picture of the distribution

07:27 that you care about. And then you can go use the models to explore all of your user data and start vibe checking against your user data and start asking questions to the models and have these models pull from the data and kind of start surfacing these patterns and the failure modes and the test cases and the edge cases that you want to care about. And then you want to do these things like generate synthetic data, do distillation, do RL. And there’s a bunch of different use cases for these evals, but really it’s just the same thing as a KPI. It’s the same thing as any metric you check on a dashboard. It’s just something you’re watching as your software is deployed in the world and

07:57 it’s getting feedback from real-world users. And in production, maturity is kind of what matters, and it’s what allows software companies to scale and deploy software that is like cost-effective for a great many users with great margins. And it comes from evolving from someone wrote a script, which is just their idea for a new feature, that then gets onto the roadmap and becomes a product because it’s kind of gone through all the stages of being hardened into something that is actually like great for the end user to experience. And this is really what we’re doing when we’re playing with agents as well, is we’re testing stuff out. We’re coming up with ideas.

08:28 We’re moving them along the maturity cycle to get to the point where we’re not just picking a model, but we’re also caring about objectives that maybe none of the models, even after all this optimization, like because we’re always going to see what the thing can do and get new ideas and want something else. Like we have not finished building software yet, even if it’s easier and faster to build software. We’re building more software than ever because we haven’t finished building software yet. And this is because when we see a thing that works, we want to know what else we could do. We’re always looking for the next thing. And so this is like, until we have kind of like, I don’t know, solved all entropy

09:03 in the world of possible things people could want to do, and there’s just nothing left to do because we’ve done literally everything and we’ve somehow remembered it all, we want more, and so this will mean we want to try new things, which will mean there’s a need for new features and new capabilities and new use cases and more efficiency and lower latency, and discovering this curve of the Pareto frontier between capabilities, features, cost, latency. This is what software life cycles are about, and it’s also what model life cycles are about. And so distillation and RL, I won’t spend too much time on this, but I think these are the families of methods you

09:35 should be aware of, which is that SFT is great. On-policy distillation from a same family teacher, if you have a bigger model that you can distill on-policy into a small model, is great. Doing it online has benefits of being better for not forgetting and remembering. It also has trade-offs of, in some cases, if the thing you’re distilling from is like too far off-policy. Like the more online you go, the more on-policy you kind of need to be. And so this is really the tension between these two families of methods. And so I personally am a big fan of offline context distillation, I think is great, especially if you combine it with on-policy teacher distillation from the same

10:06 family. So you can train data into a teacher. This teacher now knows the model, knows the stuff you want to learn, but it’s not necessarily like a great model for doing the thing because it has, you’ve broken things in the process, but you can within distribution do online distillation from this new teacher. And this is a relatively repeatable process where you can kind of create new teachers by doing context distillation or other kinds of distillation. And then this is another way you can bake them into the model. And this is generally what I would recommend to people over jumping right to OPSD. I think a lot of people have gotten very excited about OPSD.

10:37 I’ve had a lot of conversations and seen a lot of experimental results, and of my own, where it doesn’t work the way people imagine it would. It has a nice, if you look at it from a bird’s-eye view, it looks like it should just be magic. But it turns out it’s not magic. It still follows laws of physics, and like injecting knowledge into a model is hard, and there’s not a silver bullet. But I think especially in cases where you are kind of leaning on retrieval and tools and system search to get your facts, one of the easiest ways to just fairly reliably improve the model is RL. And I think RL is also the one that is most kind of Bitter Lesson-aligned with the

11:11 eval-environment duality, where you care about your metrics and you want to press a button to make your metrics go up. RL kind of is the button to do this. And some things it’ll learn better than others. But I think in some cases the things that it doesn’t do in terms of like knowledge injection or consolidation are the sorts of things that often people don’t want to, like, have hard-coded into the model anyways, because there’s different life cycles of how fast you update these things. There’s like patches and major and minor releases, and you want to be able to think about like, how can I like robustly scale the model without like putting something in

11:41 that’s going to be hard to take out. And so the stuff that RL learns generally ends up being, it’s more surgical around fluidity and harness maintenance. And so I think a very natural use case for online RL is if you’re always updating your harness, you can do online RL in the harness. And if the harness changes a little bit or the system prompt changes a little bit, like that’s okay because the RL can kind of account for these differences very quickly. And so I think those are the use cases, or like selection, you can use RL to do model routing. You can have a small model that is designed with RL to understand, okay, for the

12:12 current family of models I’m considering, like which one’s best for which task. Like this is very much a canonical, classic RL problem, and there’s lots of things like this where it doesn’t need to be jumping all the way into, I’m competing with Cursor or Anthropic, OpenAI. It’s more just about like, where does direct hill climbing of a metric fit into the model pipelines for the systems you’re building? And so then there is the extreme of this, where I think one place I see more companies going, that I think is often a lot of what I think about all day and what we’re working on, is making it easier for companies to just be very environment

12:44 native, where you want your agents to just be environments by default. You want the production to become, like you want to be able to simulate the world of the products you’re building over the company itself in terms of the data flow. And I think when you tell someone, hey, do you want to turn your company into an RL environment? Like, so you can train your model in your own company and have it be continuously improving? It sounds crazy and sci-fi, but if you say, hey, do you want like a dev and a prod? They’re like, yeah, of course. And like that’s kind of what it is. It’s like, it’s your dev version. You need to be able to deploy the thing in a way where it’s not actually breaking prod. And that is an environment that looks like the scale at which you’re

13:16 deploying models. And so whether or not you’re training all the models, like turning the full system into something that is evaluatable by default, and where collecting traces and evaluating it as part of the life cycle of maintenance is just native in the product, is the way that we see people wanting to build software systems going forward with models in the loop. Because it just means that all the things you get out of evals, you kind of get for free now, because you don’t have to like have a, oh, I’m going to have my eval version that like is like 20% of my product. You kind of want the eval surface to become, wherever agents are is where you’re evaling. Because then you can make the choice of like, okay, do I, you can collect

13:49 all your traces and say, do I need them or not? If you don’t, that’s fine. Do you want RL here or SFT there? Like, it’s fine. It’s up to you. It’s just that you want the eval to be high fidelity. You want the eval to match what you’re actually seeing in production, the same way that you want your metrics you collect in prod to be the sorts of things you can estimate in dev. And so this also raises the question of like, okay, when do you build versus buy? When do you fork? And I think it’s a very similar question to normal software, where you can ask like, okay, when do you buy a vendor solution and just use it for some kind of

14:20 software thing, versus when do you find the best open-source version and fork it and make it your thing? And a lot of it is like, well, how good is the best thing at your thing? How far out of distribution is your use case from the main thing that the thing is optimizing for? And like it is great that the frontier models are quite good at most things, but I think one-size-fits-all has rarely been the silver bullet solution for anything in software. And I think models, especially once you have all the trade-offs of cost, latency, and performance, you start realizing that oftentimes you want the more specialized thing. Like you can get almost any software cloud product in

14:53 the world from Amazon, GCP, AWS, GCP, or Azure. And yet we have a million new clouds starting every year because there’s something that is not quite there in the mainline thing, that there’s a niche to uncover by saying, “Okay, I’m gonna make the best browser sandboxes for agents. I’m going to make the best code storage for agents. I’m going to make the best like deployment life cycle loop because GitHub Actions is slow and I want to have something that replaces it.” And there’s so many of these things where just the nature of like complexity of trying to do everything at once is that companies are companies.

15:23 They have kind of one set of shareholders. They have one stock price. And it’s hard to balance things on every timeline and keep everyone happy that works there. And so like it’s hard to do everything at once as a big company. And I think to some degree this is true of the labs as well, where it’s like they’re going to make great models that are really good at a lot of things, but are they going to make the best model in the world for everything? I’m generally skeptical of this. And I think one industry I like to point to that I think has kind of fully embraced this is quantitative finance, where if you ask a quant what model they use, they’re like, “What do you mean? I have millions of models.” There’s a model for

15:54 everything because there’s so many time scales, there’s so many different applications. And I think that is one of the more superintelligence, machine learning-pilled industries in terms of embracing just like full-on dev versus prod, models everywhere, automate everything, just data comes in, decisions come out. And I think what LLMs have shown us is that this is kind of where a lot of the rest of the economy can head too, where it’s just, we still have to think about the complexity. We still have to design these systems. We still have to optimize these systems and care about our metrics. But it’s a tool that allows us to take stuff from the realm of pure statistics and time series into more general knowledge work.

16:25 But the complexity of types of model applications is still, no, there’s no reason it should be any smaller than the complexity of types of work people do, especially as models get better. Which means we do have to care about these Pareto trade-offs at every point of the curve. And so at some point we start by just using the thing that works, and then we decide, do we build versus buy, do we fork? And there’s a lot of great open-source libraries out there, and most people don’t build complex software necessarily fully from scratch above a certain scale. You kind of look at the references of what people have done before. You build on the, you stand on the shoulders of giants, and you build new software based on the patterns, whether it’s literally forking or it’s using the

16:57 patterns. Like we’re always learning from other things. And I think this is the same as like, whether you’re doing pre-training from scratch or whether you’re doing distillation or whether you’re doing RL, like designing a model for a use case is the version of building it in-house, for realizing that you need something that is specialized and custom, because you want to go beyond what the out-of-the-box thing is going to give you, because the out-of-the-box thing is not thinking about your use case as the top most important thing. And so we have this life cycle then that emerges of model development, where we want to spec, we want to know what we care about. We want to care about our metrics. Then we train our model. We use all of our methods.

17:27 We test it. We stage it. We release it. We monitor it. We see how people use it. We tweet about, hey, let me know what you like about the new model and what things it can be better at, because people are realizing, oh, it can do new stuff, but it also can’t do this other new thing, because your A gives you an idea for B, but B doesn’t quite work even though A works. And this is the way that we develop products. It’s the way that we’re developing models. It’s the way that the labs develop models. If you kind of look at any of their social media presences, they’re always asking users, hey, what can it be better at? Because they know that people are always going to have an idea of a new thing. They’re always going to find a new use case.

17:58 And that’s kind of what it is like to use models nowadays. People are complaining about cost. People want, the more that people have powerful models, the more of them they’ll want to use for different things, which means we have this knob we can turn of like, okay, we actually can just like make a thing better by making it smaller and more specialized and focused because it no longer needs to do everything. You’re paying a cost for models that can do everything or almost everything by having them need to have larger weights and be very flexible across workflows. But as processes and workflows mature, you’re kind of overpaying in that you

18:29 don’t need all of the other capabilities anymore. You kind of want your custom solution because it means you can iterate faster. You can have more control, just in the same way that people build software from scratch. You don’t, like, you’re not always going to get the exact thing you want at the right price, and you can often get something much cheaper by having something that is much more tailored to your workloads. And just like with software development, the software development life cycle is always automating itself. There’s a lot of things that are kind of routine maintenance you do in software. You have to get the docs, you have to, you ship a new SDK, you do continuous integration, you do nightly builds, and engineers have never been ones to say,

19:01 “We’re just going to do it manually each time.” Like, generally, you want to automate the process. It’s kind of like what we do in software is we figure out a thing that works and makes our lives easier, then we do it, and we do it enough that we gain confidence that it’s good, and then we figure out how to never do it again because it’s automated. And so, I think we’re in this early stage of this right now with post- training, where a lot of people are like, “Oh, post-training is really hard.” Yes, it is. It’s hard. It’s annoying. It sucks. Same as like running manual CI/CD. Like we’re going to get there. We’re working on automating the process of it because we’re poking at the edges of it and understanding the best practices.

19:32 And then once we have the best practices, then great, the world knows how to do it. And it becomes the sort of thing that becomes much more routine and mundane because we all just like have our systems that like, do you guys know how GitHub Actions work? I don’t actually know how GitHub Actions work. I know it doesn’t work very well, but it’s, this is the sort of thing where I think a lot of people are having this feeling about post-training, where it’s like, oh, I have, there’s so many things to learn, so many concepts, but like in college we all like learn about operating systems classes, if we take those classes, and like there’s so many things that are under the hood in software that eventually get abstracted

20:03 away. And I don’t know how a B-tree works anymore. At some point I did. There’s, I knew Verilog for one semester. I don’t know it anymore. There’s so many pieces of the stack of modern software that are useful to go through and then forget, because then you build on top of them and we abstract them away. I think now we’re all in post-training college right now, in terms of trying to think like, okay, what type of RL works, do I do this thing? Where we’re doing our problem sets, we’re experimenting, we’re tinkering, and then at some point we grow up and mature, and we realize it’s like, oh, we just, well, we figure out the things that

20:35 work and the things that don’t, and we start seeing what this trade-off curve looks like for when do you do it, on what data, with what recipe, and we’re getting there. And yeah, the boring version of it is just that continual learning is software development life cycles for models, and we’re building a lot of software with a lot of models in… And so I think we’re going to want some life cycle.