Why Nobody Has Solved Continual Learning

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Why Nobody Has Solved Continual Learning · Channel: LangChain · 19:09 · Watch on YouTube

Topics: AI agents, AI models

Summary

The speaker explains why continual learning remains unsolved: supervised fine-tuning (SFT) can teach a model new information but may fail when it makes mistakes, while reinforcement learning (RL) is more on-policy but can only strengthen behaviors the model already has some chance of producing. Distillation can warm-start a model for RL, though the talk describes results where RL without distillation can reach a higher ceiling on some tasks. Repeated SFT can cause catastrophic forgetting, loss of recently learned information, and degradation of general capabilities; RL avoids some of these issues but struggles to embed specific knowledge. The right balance of SFT or distillation and RL depends on how narrow or general the task is and how frequently the model is updated.

Key points

Continual learning means updating a model repeatedly from collected traces, rather than relying on a one-off SFT or RL run.
SFT can teach behaviors or reasoning traces that the target model might not produce on its own, but its off-policy data may not show the model how to recover after its own mistakes.
RL is on-policy and can improve successful behaviors the model already samples, but cannot readily learn behaviors outside its current distribution.
Distillation can warm-start a model with useful examples before RL, but the speaker describes experiments in which RL without distillation eventually reaches a higher ceiling on some tasks.
Repeated SFT can cause catastrophic forgetting, faster forgetting of previously learned post-training information, and declines in general capabilities.
RL tends to cause less forgetting and task interference, in part because an implicit KL penalty keeps updates close to the base model.
RL is not an effective general method for adding specific new knowledge that the model is unlikely to generate or encounter in its rollouts.
For narrow, stationary tasks, repeated SFT on high-quality traces can work well; broad, agentic tasks are much harder and may require frontier-lab-scale training.
Teams need to choose the SFT/distillation-to-RL balance based on where their task sits between narrow and general.

Chapters

Start	Chapter	Summary
00:03	The continual learning loop and its training challenge	The speaker introduces the feedback loop from agent traces to evaluation and training, emphasizing that continual model updates remain difficult.
02:10	The trade-offs between SFT and RL	SFT can teach new traces but may leave a model unprepared to recover from its own mistakes, whereas RL can only reinforce behaviors the model already samples.
06:18	Distillation as a warm start for RL	The speaker discusses using distilled traces to give a model a starting chance on RL tasks and describes results where RL without distillation reaches a higher ceiling.
10:07	What breaks under repeated training	In continual learning, repeated SFT can cause forgetting and capability loss, while RL is more stable but poorly suited to instilling specific knowledge.
15:26	Choosing a strategy for the task	The speaker contrasts narrow tasks, where SFT may work well, with general agentic tasks that demand more extensive training, and concludes that teams must balance SFT and RL.

Quotes

the train box is like not solved at all. — 01:04

This is a big failure mode of SFT. — 04:12

This is called catastrophic forgetting. — 11:10

RL is not enough of a general solution to bake in our, like, continual learner. — 14:23

you just need to be very smart about, like, the ratio of SFT/distillation to RL that you do. — 19:09

Concepts and entities

Continual learning, SFT, RL, distillation, catastrophic forgetting, DAgger, Qwen 4B, Fable 5.1, ECHO, Cursor, Composer, Anthropic, OpenAI

Full transcript (original language)

00:03 Very excited to go through some of the things we’re thinking about from particularly the research side. But in terms of how we update the model weights for a model, like, in terms of owning your intelligence, that’s one of the big things you can do. So it feels like there’s a lot going on, and I’m going to try and, like, give a big context dump of at least, like, what we’re thinking about, what we believe even the frontier labs are thinking about when it comes to, like, post-training and even earlier stages of training with your models. So this is kind of like the loop that, you know, everyone refers to.

00:33 This is the loop you see in all the, you know, own your own intelligence slides. It’s essentially like, okay, we have an agent doing some sort of task. We are able to extract traces for that task. We are able to then look through them and see what the model is doing right and what it’s doing wrong. And we’re going to turn that into some form of an eval. Now, this might look like an environment with a reward function, as we see in RL. It might look like an eval that turns into, you know, a perfect gold trace that we can then SFT on, supervised fine-tune on. Whatever it looks like, there is some way to

01:04 determine good from bad in the traces that we’re generating. And then, you know, there’s this box, which is like this very nebulous, like, train. Once we’ve trained, we’re hoping the model gets better and we deploy it. I think the thing I want to like hammer home today is that no matter what anyone tells you, like, the train box is like not solved at all. It’s really, really, really hard, and everything that you do has problems and trade-offs. And so I guess for everyone’s context, I’m talking about the continual learning regime. So if you’re going to own your intelligence, like, you don’t want to just do

01:36 a one-off SFT run or a one-off RL run. You want your model to continue to get better from these traces that you’re collecting. Like, that’s the whole point of having this feedback loop, is that it is a loop that you’re consistently running. It’s not a one-off thing. So I guess today, like, I’m going to talk to you about why you can’t just continually SFT. You can’t also just continually RL. Like, there is no perfect solution. I’m going to try and, like, sketch the trade-offs and the, like, give you an outline of all the different methods that we can use. So yeah, maybe, like, a place to start here is, like, trying to summarize the literature

02:10 around RL, SFT, distillation, etc. So everyone probably has this vague idea of SFT as being able to install new information in a model. So I can take a gold trace from, you know, Fable 5.1 and give it to Qwen 4B and SFT on that, and, like, Qwen is going to get new information and it’s going to get, like, I guess, reasoning traces or paths that it never would have sampled by itself. That’s a great thing about SFT. RL can’t really do that.

02:41 RL is limited to what we say the support is of the model. So if a model can sample a particular rollout, a reasoning trace, however you want to call it, we can upweight the likelihood of that reasoning trace if it’s good. So RL can only take things which are in the support of the model. SFT can kind of make the model do whatever it wants. Now, there’s a lot to break down here because, like, from that, like, very brief description, it seems like SFT is, like, exactly the thing that you’d want to do. Like, I can get my Qwen3 4B to act like Fable 5.1.

03:11 I just get enough SFT traces. The problem comes here because SFT is a very off-policy process. So the data that I am learning from, yes, it’s got a very dense signal. Like, every token, I’m getting a new reward about what the correct next token is, according to Fable 5.1, but it’s very, very off-policy. And so I fine-tune my Qwen3 4B on this and then I go deploy it in production, and, you know, I hit this production edge case which isn’t even out of distribution. It’s just not in the training distribution. Like,

03:41 I haven’t seen this exact example before. And this off-policyness means Qwen3 4B is orders of magnitude smaller than Fable 5.1. So let’s just say it makes, like, one mistake, like one misstep. We say it’s solving a math problem. The problem with off-policyness is that Qwen3 4B has never been trained on Qwen3 4B’s mistakes. Like, it’s only been trained on Fable 5.1’s rollouts. And so that means that as soon as it makes a mistake, it’s got nothing in its training data which shows how to get back onto the correct reasoning path.

04:12 And this is a big failure mode of SFT. This was first kind of pointed out in the old traditional RL literature. There’s a paper called DAgger that I highly recommend people read. Gives a lot of intuitions about this, but it’s a reason why we can’t just, like, kind of jack up SFT unless we can literally SFT on everything, which is kind of the idea of pre-training. RL, in, like, kind of the opposite end of the spectrum, is extremely on policy. So, we’re only ever taking samples from the model, Qwen3 4B, that we intend on using in production. And so let’s say, again, we’re in this, like, kind of maths

04:46 environment. Like, we are going to upweight Qwen3 4B’s probability of doing good rollouts, so when it gets a positive reward, and then we’re going to downweight its negative rollouts. But there is a limit here, right? Like, let’s say that we’re trying to solve, you know, an IOI problem from 2026. Like, it’s very unlikely that Qwen3 4B is going to sample. I probably shouldn’t say that now ’cause like even the 4B models are, like, possible. It’s possible for them to do IOI problems, which is kind of crazy. But, like, you know, if it’s a very hard math problem, it’s going to be very unlikely

05:18 for that small model to be able to sample a correct answer. And if we can never sample a correct answer, we’re never going to be able to update it. So there is this trade-off, like it’s incredibly on-policy and like, you know, we’re not damaging the model in this way. The model is going to know exactly how to recover from wherever, at any point in time, because it’s always coming from the model. But, you know, we have this ceiling of, like, we can only upweight things which are already in the distribution of the model. And so that’s a lot of information, but hopefully, like, now I’m starting to instill into you, like, okay, there’s a lot of these trade-offs.

05:48 I can use RL to improve my model, but that is very, very dependent on where my model starts from. SFT is not really like that. I can take some really good gold data, and no matter where my model is sitting, like, training the model on that gold data should improve according to the data that I’m training on. But RL’s not like that. But of course, SFT has, like, the problems that I just talked about. So this is, like, the starting point. I want to talk now a little bit about, like, some of the ways that we’ve been

06:18 thinking about, like, when to do SFT/distillation and when to do RL. And I think this is, like, a very interesting question in general because you know it allows you to predict what, you know, the Chinese open source lab should be doing, like how much distillation should an open source lab be doing versus RL? And this was one really interesting result which we replicated in a few different ways. So as I just told you, you can’t use RL if it doesn’t, if you don’t have any probability of sampling a good rollout in the first place. So what a lot of people do and what we believe, you know, the open source labs to do,

06:51 is they will take a model after pre-training that’s probably got a very low likelihood of sampling anything useful on any of the RL environments that we care about. So distillation is then used as a way to warm start this model. So we’ll take some reasoning traces and some trajectories from, you know, a frontier closed source model, and we’ll throw those in at the end of pre-training or in kind of like mid-training or continued pre-training. So the model sees some examples of how to do this task correctly. And this looks exactly like SFT. Like, this is literally just SFT. We’re just doing next-token prediction on these traces.

07:24 That’s what we believe distillation is used for at the open source labs. It’s a way to get a nonzero pass at, you know, k for your particular RL environment, and then you do your RL from there and you watch your model climb. And this is typically like the process that’s used, and you know, like we see many companies going through this process. So they’ll have, you know, they have, like, a bunch of very complex RL environments. They want their model to get better at them, but the model’s starting off at a really, really low pass@k. Like, pass@256 might be, like, really, really low, and they just don’t have the compute to be able to run, like, that

07:58 big of a group size for that long to watch the model climb. So they’ll do a bit of distillation on, you know, some closed source traces and then do the RL after that. To get to the point, I think what this graph is showing, and again, like, we’ve replicated this in many ways, is that, really interestingly, if you do, like, any meaningful amount of distillation in order to get your warm starting, and then you do RL properly and you do it until, like, you’ve really, really plateaued, basically you start off much higher after you’ve done distillation. Obviously, that’s the whole

08:29 point of doing it. But RLing on a model that hasn’t been distilled at all ends up getting much better. So you have this kind of, like, curve where the distillation curve starts higher but it plateaus much lower than the model that’s been RL’d. And this was a little bit surprising, and we thought this was a quirk of the initial setup that we’ve done, but this turned out to hold across a lot of things. And it’s very dependent on the type of task that you’re learning as well. So it seems like the RL models, like, reach a much higher ceiling in, like, calculation or math-based tasks, purely because like I think basically distillation is constraining the like entropy,

09:04 you can think of, of the model. So we’ve kind of, like, tried to apply this in, like, you know, we train a lot of models for customers. Now we’re actually really thinking about carefully about whether we do any distillation at all to start off with. Like, now we’re much more pilled on, like, if we have the compute, we’re going to RL, like, all the way. It means you have to care about different things, like, for instance, you have to care about the curriculum of your RL task. Like, if you’ve got some task that, you know, you have a good pass@k, and some task where pass is, like basically zero, you kind of have to order this curriculum. It’s another thing you

09:35 have to care about. But, like, you know, if you get to a higher point at the end of the day, then it probably is worth caring about it. Another thing that, like, I kind of want to talk about in terms of, like, results that we’ve seen recently is this idea of, like, I guess, yeah, continual learning. So right at the start I mentioned that basically if you’re doing a one-off like, you know, big training run, these things are kind of solved. Like, you can use SFT, your model is going to get better. Like, yes, there’ll be things outside the distribution

10:07 that it won’t go well on, but like, you know, you’ve got a big example of, like, Fable 5.1 tasks or rollouts, like, your Qwen model is going to get better. Same with RL. Like, you collect a bunch of environments, you do the RL, the model hill climbs, like, you got a better model. I think what’s really interesting to me is, like, the continual learning regime, where you’re doing this process, like, not once and not, like, 10 times, but probably, like, hundreds if not tens of thousands of times. Like, your actual, like, weight update is happening, you know, at least multiple times a week.

10:38 I think, like, this is the most interesting world to live in, and I think it’s the world that, like, a lot of companies are going to have to shoot towards because, you know, they need some sort of moat against Anthropic or OpenAI. And the thing that breaks down here is, well, it’s a few things. So the first thing is like, if I keep SFTing on, let’s say I have this, like, you know LLM that’s a legal associate, and I’m trying to train this legal associate on all the new information it’s seeing. So, you know, it’s learned new things about this case, and it’s learned that this person has joined as a paralegal, and it’s learned that this information is stored

11:10 here in this firm, and so on. So again, like, I can kind of, like, you know, do a big fine-tune with SFT. So we use something called self-study if we want to learn, like, knowledge. Do a big fine-tune once, that’s fine. If I repeat that process over and over and over again, I find that multiple things happen. The first thing is I forget things in my base distribution which are useful to the task that I’m currently doing. So this is called catastrophic forgetting. Most people have heard about this. I will forget not only, like, you know, English literature from the 17th century, but I will actually start forgetting law stuff as well at some point, even though

11:42 that law stuff is supposedly helpful to me and my legal associate agent. The second thing is I will forget stuff that I’ve learned after pre-training. Like, you know, let’s say the first day I’ve fine-tuned on, you know, this particular case, and then, you know, 100 updates later, I fine-tune on a completely different case. What we find is that the model will forget the earlier cases, and at a much faster rate than catastrophic forgetting happens in the model in general. That’s really interesting. The third really bad thing is that the model’s general capabilities will degrade. So if you start testing it on, you know, instruction following, maths, reasoning,

12:16 all these kind of, like, general tasks, we see that this really fails in the iterative regime. So it’s clear that, like, you know, self-study plus SFT is not this like silver bullet to continual learning, and, like, it doesn’t work when you have this very, very broad task. I will caveat this with saying that this does work when we have, you know, a very constrained task with a stationary distribution that’s not changing. Like, I can keep fine-tuning on that. I can keep generating gold traces on that, and my model will keep getting better. Like, but for very, very general tasks that are non-stationary,

12:49 like new information is coming in as well as new capabilities, SFT on the self-study stuff doesn’t really seem to work. So you might say, okay, what about RL? RL, I will say, doesn’t mess with the model in the same way that SFT does. So, you know, we don’t see that much catastrophic forgetting with RL. We don’t see, you know, degradation of general abilities. And we see little interference between tasks that we’re kind of RLing on. Basically, the reason for this, like, you can read all the literature about it, boils down to the fact that the model has, no matter what RL kind of method you use,

13:21 basically this implicit KL penalty. It’s like tying it very closely to the original base model. So when we RL, we just, like, make very, very small nudges to the model. But the problem with RL is, like, it’s very hard to, one, bake in knowledge. So again, as I said, we can only change the behavior of a model in RL if the model is likely to sample something. A model is very unlikely to sample something like, you know, Jenny has joined as a paralegal in team XYZ. Like, some very, very specific knowledge is very hard to, like,

13:52 engineer RL environments to bake in knowledge, and secondly it’s very hard to like induce that in any way through any sort of efficiency penalty. So you might imagine if you had the world’s best efficiency penalty, you could incentivize the model to, like, learn some behavior instead of having to search for it each time, but we’ve never seen this happen with RL. Like, the model will get better at searching and it’ll get better at grepping and everything, but it will never be incentivized enough, even with the highest efficiency penalties, to say, okay, I’m going to remember this information for next time, simply because of the mechanism of how RL works.

14:23 So I think, like, that’s the main thing that you should take away from this. Basically, in the iterative regime, a lot of these things break down, and RL is not enough of a general solution to bake in our, like, continual learner. There’s been, like, a lot of approaches, like ECHO and, like, you know, even Cursor recently with Composer. They tried this thing where they adapted on-policy self- distillation to, like, essentially provide hints, like dense-level hints, to the model in doing RL at kind of, like, rollout time.

14:54 So they’re trying to combine the best of both worlds here, essentially, where they go, okay, we’re going to have an RL reward, we’re going to have a self-distillation reward where we’re actually trying to provide new information to the model. I think the general take with, like, self-distillation even is very, very hard to get to work because it naturally introduces bias into your RL algorithm, and LLMs, by the time you get to RL, are very, very, like, anti-bias. Any sort of bias will mess with them, but I’m happy to talk to people about that later. Yeah. So, to kind of take away… how much time I got?

15:26 Okay, we got a little bit of time. So, to take away, like, I’ve kind of dumped a bunch of stuff on everyone. What does this really mean when you have this particular task that you’re doing with some form of LLM or agent, and you want to, like, own this continual learning loop? So, I think everyone’s kind of aware that if you have one task and it’s extremely narrow, then you’re kind of, like, sailing. You’re cheering. You’re really happy, like if it’s, you know, writing clinical notes so I’m dumping in a transcript and expecting some clinical notes with a particular

15:58 prompt structure out, or you know, I’m doing, you know, voice extraction, so I’ve got like you know speech-to-text, LLM, text-to-speech kind of setup. I’ve got these like phone calling customer service agents. That’s another example of a task that is like really narrow and theoretically my rules and distribution in general is very stationary. So I can, you know, the best thing to do probably here and like don’t quote me on this, is, like, take the best model you can find, ignore the terms of service, and generate

16:28 like hundreds of thousands, if not millions, of traces and SFT on that. Like, again, don’t quote me on that. That wasn’t legal advice or anything, but the idea there is that you can just take these gold SFT traces and SFT on them, and you’re going to asymptote towards that gold distribution, and in the infinite data regime, you’re going to get there. That’s great. Then, you know, the world got more agentic. Someone decided to give LLMs tools. We started to, like, let them roam free a little bit more until like basically, like,

17:00 every LLM harness was this like kind of universal Turing LLM machine that allowed them to do whatever they want on whatever sandbox they wanted. And so, in this case, like, things are a lot harder. As an example, people that have tried to own their own intelligence at this layer have been, you know, Cursor, Notion, Harvey, Lovable, like, a bunch of these companies that are realizing that when your task is general enough, you basically have to turn into a frontier lab in order to own your intelligence. So this means, you know, post-training on a bunch of environments that you yourself

17:33 create, as, like, we’ve talked about before, but also, like, doing the earlier stages as well, the mid-training, which again looks like SFT. So from that perspective, like, yes, it’s doable, but it’s very, very hard. I think the last thing I want to, like, convey is this continual learning loop, like, we’ve already solved. Like, it kind of sounds like I’ve said that this is, we haven’t solved it. I mean, we haven’t solved it at a really granular level when you’re operating or you’re updating your model, like, you know, multiple times a day.

18:04 I think if you look at what the frontier labs are doing, like, they are running a continual learning loop, just on a very slow cadence. So, you know, they train a model, release it to the world. Like, some will look at the exact user data, some will ask for user feedback. They aggregate all this and look at all the things the model can’t do. Great, we’ll turn that into RL environments. You know, if we get zero on those RL environments, we’re going to, like, use privileged information or some way to construct these great trajectories. We’re going to mid-train on them, and then we’re going to do RL from there and warm up from there. So I think if you look at this loop, it’s already running and it involves both SFT and

18:38 RL. It’s just, at that scale, we don’t have to worry about catastrophic forgetting, like general capability degradation, all the things that I mentioned earlier, the breakdown. Yeah, I think that’s the last thing I wanted to convey. But yeah, most of the people in this room will live kind of in the middle. Like, you have a task that is, like, not quite narrow, but also not quite Cursor. And you will be able to come up with some way to bake both the knowledge and the capabilities that you want to, like, continually get better at into that model.

19:09 You just need to be very smart about, like, the ratio of SFT/distillation to RL that you do. And I think that’s the most important place to start thinking about this loop. And so technical teams, post-training teams, etc. Like, that’s the place to start looking and thinking about, like, where on that spectrum you sit. Cool. Thanks, guys.