Open by design: Together AI x MiniMax — Lightning talks

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Open by design: Together AI x MiniMax — Lightning talks · Channel: Together AI · 42:11 · Watch on YouTube

Topics: AI infrastructure and hardware, AI models, Local and open-source models, Multimodal models

Summary

The event explores how open-weight AI models can be put into production, with Together AI outlining the infrastructure stack and MiniMax discussing its multimodal models and production cost trade-offs. Speakers emphasize that open models offer control as well as potential cost and performance benefits, while requiring careful optimization of inference, memory, caching, and concurrency. MiniMax presents examples involving video generation, long-context text, and domain-specific post-training. Together AI’s Max then introduces score centering, a technique intended to stabilize off-policy reinforcement learning when the trainer and sampler differ.

Key points

Together AI describes an open-model stack spanning model weights, inference, gateways, application harnesses, and tools.
The speakers identify cost, performance, and control as major reasons organizations adopt open-weight models.
Together AI says its platform serves more than 200 open-source models and provides globally deployed compute and inference infrastructure.
MiniMax presents openness and multimodality as central to its approach, with open-weight models across text, video, and music; its speech model is noted as supporting private deployment.
MiniMax M3 is described as supporting up to one million units of context and using sparse attention to reduce the cost of processing context.
Production cost and quality depend on operational choices such as model precision, prefill and decode optimization, caching, and concurrency.
MiniMax and Together AI describe inference optimizations that include attention and kernel changes and offloading some preprocessing from GPUs.
The speakers describe post-training and reinforcement learning as ways to adapt open models to specific industries and tasks.
Max presents score centering as a method to account for trainer–sampler mismatch and improve the stability of off-policy RL.
The talk reports that score centering can be combined with importance sampling and shows promising results in experiments, including larger-model training.

Chapters

Start	Chapter	Summary
00:01	Welcome and the rise of open-weight models	The event opens with an introduction to growing demand for open-weight models, especially in Europe.
02:07	Why teams choose open models	The speaker discusses cost, performance, and control as key reasons for adopting open-source models.
03:08	The open-model production stack	Together AI outlines the layers needed to serve open models, from inference and gateways to applications and tools.
05:50	Use cases and post-training	The talk surveys coding agents, financial-services workflows, content generation, and reinforcement learning.
09:09	Together AI infrastructure and inference	The speaker describes Together AI’s model platform, compute, and work to make inference efficient and reliable.
11:16	MiniMax and open multimodal models	MiniMax introduces its background and its focus on multimodality and making model weights openly available.
14:09	Video generation and the M3 model	MiniMax discusses an open-weight video model and introduces M3’s long-context capability and sparse attention.
17:18	Cost-control levers for production	The speaker explains how open models expose choices around memory, precision, caching, decoding, and concurrency.
22:20	Inference optimization and industry adaptation	MiniMax describes Together AI’s serving optimizations and a post-training example for slide generation.
26:09	MiniMax takeaways and data residency	The talk summarizes the roles of model labs and the ecosystem, including European data-residency needs.
27:52	Introduction to reinforcement learning research	Max introduces score centering and frames the challenge of scaling RL while controlling costs.
30:05	RL compute costs and instability	The presentation explains why scaling RL can improve results but also creates high compute costs and unstable runs.
33:24	Trainer–sampler mismatch	Max explains how differences between the model generating rollouts and the model being trained can undermine RL updates.
36:08	Score centering and experimental results	The talk presents score centering as a correction for persistent distribution mismatch and reviews its results with other methods.
41:38	Implementation and conclusion	Max notes that score centering can use only the sampler’s top 128 tokens, then summarizes the method and its compatibility with importance sampling.

Quotes

the core you know reasons we’re seeing around uh you know folks moving to open source models or cost performance and control. — 02:07

making intelligence accessible for everyone — 13:35

which tasks should I send to the best model — 17:18

The idea that we proposed is called score centering — 36:43

Concepts and entities

Together AI, MiniMax, MiniMax M3, H3, Together Link, score centering, reinforcement learning, importance sampling, DeepSeek R1, Qwen 3 0.6B

Full transcript (original language)

00:01 Welcome everyone. Thank you very very much for being here. Um you know a real pleasure to get us kicked off today. I think this is our fifth uh event that we’ve run like this. So um we titled this session open by design. You know over the last 3 to 6 months we’ve seen incredible demand specifically in Europe um to go into the open weight world. So it’s no longer a question of like do we don’t we want to use open weights is how do we actually do it? And today’s session uh you’ve got um a number of speakers from both together uh our

00:32 partners at Miniaax and of course one of our customers to be in deal. I’m going to introduce our first speaker in sung who’s VP of customer success uh for together. Uh I’m Sar Tropathy. I lead customer experience at together. Um that is really just sort of means everybody that works with our customers directly to make sure that they have strong business and technical outcomes. um that solution architecture for deployed engineering support uh and just making sure all of our customers are successful. Yeah. So I I just kind of

01:03 want to started start by kind of discussing why we’re here today. Uh as James mentioned, there’s been quite uh a lot of uh noise, positive noise around open models uh the last uh several months and uh and and I just sort of was reading over the weekend and found this Mozilla um you know sort of open state of open source report and this reports across all open source technologies. Uh what was fascinating is that seven of the top 10 models by token volume right now are openw weightight models which is super super exciting and quite different

01:34 than what you would have seen a year ago or even 9 months ago. Uh and so open source is really taking over. What’s super exciting is that our partners at Miniax are right inside that openweight uh wave um with um with their bleeding edge models that are solving all sorts of amazing use cases that you’ll hear about today. uh and here at together we run that inference layers that take that inference layer that takes these models openweight models into production. Now why do people select open source models? Just out of just get a quick show of hands, how many of you have uh interacted with or deployed an open

02:07 source model uh yourselves? Okay, cool. So um fortunately then I won’t have to explain this too much but the the core you know reasons we’re seeing around uh you know folks moving to open source models or cost performance and control. cost. One example I’ll give is here in Europe recently with an enterprise we had a we had a customer save 98% uh using an open source model versus uh a model deployed on open AI uh performance we seeing anywhere from 3 to 5x gains on open source models versus closed source

02:37 models especially when you fine-tune them specific to use cases and on the point of fine-tuning control is a massive massive aspect specifically here in Europe that we’re hearing uh our customers and the community ask for uh uh around uh around models which is shifting them toward openw weight models. Now what does the openw weight model stack look like? We’ve thought a lot about this and you know with the with the nearly million developers that leverage together AI on a yearly basis and the hundreds of customers that we

03:08 work with, we’ve sort of designed this construct uh you know in our heads of the stack that makes up serving open uh open source models. And it’s really all the stuff that the many of you that raise your hands uh probably use and have seen on a day-to-day basis. The first is the model itself. Um so so you know what you know the actual model weights uh that um you know that sort of make your responses great. One key critical layer here is the inference layer itself. Uh of course this is where together sits uh in serving these open

03:40 source models. Uh I’ll just you know anecdotally say two years ago when I was not yet at together AI I did not understand nearly how complex it was to actually serve open source models or ser serve models for inference at all right there’s a performance element there’s a reliability element there’s stability element this very very complex layer which is why you’ve seen you know folks like together emerge in the market to take this burden off of you the consumer um from actually hosting the models. Uh the next is gateways. So um this is the

04:13 layer that that really decides whether it’s which provider or which model is actually serving the end user request. Uh this is a area that’s really popped off in the last uh you know in the last year or so. You saw open router got acquired by stripe for this amazing technology that they’ve built. And we see more and more routers coming up uh and gateways coming up in the um in the field um you know from from from great contributors in the open source community to just make it easy for you all uh the community to leverage

04:44 whichever models that you want and not really have to think about where uh the selection of these models. Uh the harness layer. So this is the application layer. I mean, you know, many of you have probably used things like open code, things like cursor, uh things like cloud code. Um super exciting, uh to also share that, you know, a lot of these sort of closed source variants of uh front-end applications are now able to interact with open source models. So one thing that we’ve built here at together uh is called together link uh which actually enables

05:17 developers to use models like miniax m3 um within cloud code. So if that’s something that you guys are interested in leveraging open source models within a closed source interface that’s something that u you know that we can absolutely chat about and I think something that in the community people don’t know a whole lot about and then of course there’s the tools layer uh the final layer of the might stack um which uh which is your MCPs your skills uh and the things that know exactly sort of how to do specific tasks um you know within uh within your open source stack. This is the might stack. Happy to chat more

05:50 about this, but this is sort of our take on everything that you need to run open models. Uh and uh and and fortunately uh the AI overload lords have gotten good enough to sort of do all this for you if you really don’t want to think about it. What are we seeing uh in in terms of use cases for open source models? Um you know, unsurprisingly, the the biggest is coding agents. Uh I I think most uh uh even semi-advanced uh sort of software development shops these days are um are

06:20 writing more code through agents than uh through humans which uh which is likely the right way to do things. How many of you you have used co-working agents? I’m just curious. So this is another one that we’re seeing quite a bit and being that we are in one of the world’s largest financial hubs today. I’m based in San Francisco by the way, so it’s I’m uh it’s it’s nice to nice to be in London. But what this is a use case that we’re actually seeing quite a bit of in financial services. Uh there’s so much sort of excelling and various different application work to be done in different financial systems that

06:53 training co-work agents to go do this um with really really smart open source models is something that we’re seeing quite a bit of. Uh and then of course voice content generation. Um, the likes of folks like Higsfield are are making uh incredible incredible movie productions, not just, you know, cat videos pretending to be Donald Trump, but like true uh true videos uh and true movies um to uh uh to to, you know, to sort of put out there using incredible open source models. Cool. Okay. So another point uh I think

07:28 that would be interesting uh for you guys to know and to sort of speak about is post- training in RL. And this is something that you’ll hear uh another talk uh from my colleague Max about this evening. Uh but post training in RL is I think you know just as in 26 2026 was the year or is emerging as the year of open source models. We see the sort of late 2026 and early 2027 to be um you know the time that’s going to sort of see the proliferation of post- training and reinforcement learning. Uh this is really the sort of basis on which uh we

08:04 believe um companies are going to become differentiated long term. So, in a world where there’s an amazing new open- source model every day, um, how do you as a company and you as a development shop, you know, stay differentiated? And this is something that we’re spending a lot of time talking to our friends at Miniax about as well as is actually helping our customers do reinforcement learning to control the quality, speed, control, and stay differentiated um with their offerings. And this is something that uh that just another sort of area

08:36 we love um giving back to users and helping them um you know stay sort of at the forefront of um of their of their domains be it financial services or healthcare or whatever it may be. So I won’t share too much here because you’re going to hear a lot more about uh you know from the expert but this is another very interesting uh space to watch. uh as uh you know as you may know uh together hosts many open source models um you know various you know across language models across uh voice models uh we have many different partners in

09:09 the open source space uh and uh and you’ll hear and you’ll hear directly from our uh our friends at Miniax about them as well today so I’ll just give you a quick quick intro into the together platform I won’t spend too much time here let you guys sort of ask questions later later but Um as you sort of heard themes we you know we provide our AI infrastructure as a service. We have compute deployed globally and serve 200 plus open source models on this platform allowing you to do reinforcement learning serve these models shape these

09:40 models giving you a rich developer experience on top to sort of manage all of this. Our compute I think one of the key differentiators of together comput is owned and operated by us uh globally. So through uh through owning compute directly through having joint ventures uh with f folks globally and through leasing some compute we do uh you know retain quite a bit of an advantage on getting you the compute that you need when you need it. Final point I’ll make perhaps the most interesting thing that we uh that we

10:11 spent most of our time doing with customers uh is making inference cost effective for the community. And so uh if you think about what this takes, it’s really sort of twopronged. Uh the first is optimizing the inference engine itself. This is these are the things that you don’t want to do yourself when it comes to inference. uh things like training specular decoding models, doing prefix caching, disagregated serving uh between prefill and decode, structured outputs, function calling. All of these different uh aspects of the inference stack that are quite complicated to to execute

10:43 yourself, we sort of do uh for our users and we partner with our uh with our developers uh as well as uh our partners and customers uh to do for you. And then of course there are the the there are the other aspects of just deploying a distributed system which um has never really been easy um and and continues to be a hard problem to solve. Uh doing autoscaling, queuing, high availability, multi-region deployments, uh deploying context pools, things like this. These are all things that between our software, our infrastructure, and uh our

11:16 for deployed teams, we’re we’re happy to help our customers with. So um with that I will pass it on to uh our friends at Miniax. Rio >> my name is Rio. Um I’m heading up the Minimax developments of course Emia. Uh we do have a team here. Some of the members like Andrean Jazz and Elliot feel free to grab them and everything that you interested about us you can feel free to ask. Um I think the content of today um is about how to run um open

11:51 models in production. So when you think about the production then you need to think about all the attributes where if you are just calling API wouldn’t be worried about. So it’s all the things that uh a open model enables you to do uh and to fine-tune to the qualities that you like. And I think ultimately the goal is to achieve a balance between quality and pricing being cost effective. Um what minmax is uh you’ve heard of a lot about the names minmax and doing uh the customer cases by um by sang here and uh just very quickly uh we

12:29 are a frontier model lab so established four and a half years ago. Uh we are the Asia’s first uh lab working on foundation models. before Chachibility was launched. Uh and over the years uh we’ve made uh our presence across the world and actually we already been uh listed on Hong Kong stock exchange. Most of our revenue comes from outside of China and we we’ve been pivoting from consumer uh services to enterprise services and last quarter report

12:59 disclosed this number. So over 60% so a majority of our revenue now comes from uh enterprise customers. So um and I think two key words about minmax uh is uh is first one is multimodality uh that is by design. Uh so right from the very beginning our interpretation of achieving AGI is actually simulation of the real world. You can’t do that without really see uh de being really good at not just the text model but also the video and the audios everything that

13:35 encompasses how you would receive and perceives information uh in the real world and second is about openness right so most of our model with a caveat that uh the speech model right now supports private deployment but everything else so ranging from text model to video model to music model even open weights so where people can download and run them. We we are doing this uh because that’s in line with the mission of the company. So uh making intelligence uh

14:09 accessible for everyone and that’s why uh that’s why we’re doing this. Uh I want to use this case to show you that the uh frontier video generation and open weights model are not something there’s a trade-off you need to choose between them. So this is the very first time that the market has a sort video generation model which is also open weights. um it’s achieved number one ranking but the ranking is something that the community work continuously to

14:39 build on with the post training uh etc. So for example like one of our other partner FA developed a fantastic live streaming model based on uh the the ways that we’ve released but H3 is a model that can generates understands and generates text video and audio in one pass. So uh let me show you a quality uh of this by a picking duck. So um any of you have tried picking duck before

15:10 great then uh so hopefully this one plays or not? So it should be playing we’ll have to skip unfortunately no ping duck tonight. So we we we have loads of uh you know examples where you can search on YouTube. Um so that was uh the the video model S3 but I think the crown of the jewel is our flagship model M3. So it’s a textbased model. Um it can intake up to 1 million of contextual information. Uh

15:43 it can generates coding uh etc. But I think the point about how uh economically this model runs is something to do with how the model pays attention right so uh when when a model runs um these context information is where is mostly expensive and we’ve developed the system such that uh it will do a cheap scoring across all the contextual trunks and we’ll be only

16:13 paying attention to those selective ones. Right? So that’s we call it the sparse attention scheme and with this selectiveness then effectively it can lower that uh running of it to a 5% level. So compared to the previous generation and it is a foundation of all the cost effects uh that this model uh builds on. So why pricing would matter? So um currently um minimax M3 it runs

16:44 you know normally at $.3 that’s something like5 to 20% of all the uh top close models um and not by our words but by uh Gilmo he runs Verso and he did say upon our model launching uh when he was testing against his own benchmarks that it is the top performance quality on par with some of the best models right behind the top uh frontier close models but 10 times cheaper. So

17:18 the question now uh stops being uh when you’re running production stops being so uh which model is the best model but now you’re asking so which tasks should I send to the best model right and at the end of the day a product system may realize that it’s a minority uh of the tasks and and that is why uh the scalability uh will be achieved using open models um and I think um saying by saying linking open model is cheaper would be

17:53 the wrong message and wrong lesson to go is really the about the the controllers right so a closed model it gives you an API it’s a simple uh price of the uh consumptions and but everything inside it uh how the model consumes the memories how they cue your request everything is closed and that decisions has been made for you but an open model opens us those layers those levers where you can fine-tune and you can walk through the partners uh like together AI

18:27 to to together to achieve the optimal on that uh cost effectiveness curve for your specific use cases and I’ll be quickly going through like some of the levers to show you like there’s real cost savings can be delivered there so lever number one is memory u memory is most important thing because uh you will probably own or rent uh graphic cards and those will be some of the most expensive things you ever owned. [laughter] Um and therefore uh deciding

19:00 like what precision level of the models that you run is very critical. So if you lower that memory is needed. So for example, if you half half it um the the precision level then it takes like 50% less of the memory but you have to uh bear in mind that lower precision does mean like a reduction in the quality of the model outputs. Therefore you need to balance uh the cost considerations of the model uh quality versus the cost of running it on your graphic card. Right?

19:36 So um the there’s some point of the curve that you need to uh consider but that choice of uh determination would be yours. Uh lever number two and number three are really I think uh both sides of the same coin. So we’ve got prefill and decode. So basically uh how uh a model reads uh and writes right. So um you have the levers to to fine-tune and for with minimax sparse attention as I mentioned. So uh you can really like jump uh when when in the contextual uh

20:13 readings. So therefore uh achieve a lower percentage of the token to first uh time arriving and uh you can also do things like uh caching right. So only stores uh the information temporarily. So uh in instead of going through the whole uh contextual reading again and in a lot of the cases contextual reading is very expensive to run right when a model takes your input. you actually go through all the back history and

20:46 documents and uh contents that you have talked through and if you do caching well uh that low that cost can be significantly lowered. Therefore, one of the most important metrics in inferencing is that cache heat rates. How you measure a platform uh the quality uh of the caching actually determines how you run uh your real real production system cost savings. um and things like the and decodings or when the rights or the output speeds also

21:18 matters. You can do some lowlevel coding optimizations like on the kernel level to improve that uh speed as well. Um level number four and this is some sometimes like get overlooked because it’s kind of like a um product decision wrapped in the infrastructure layer. Uh so like there’s some benchmark numbers for example so when you run minimax M3 on just four cards and depending on how

21:48 many concurrent users you send right so if you just send four concurrent users to those four cars and you get really fast speed like 200 tokens per second per user. But if you decide to send more concurrent user to that then each of the user will get less of the speed but the total output of that generations is higher. Therefore the per token cost of running is cheaper. Right? So you need to think about the kind of use case and applications you run for coding

22:20 assistant agent then you might decide it is more critical the speed matters. Therefore, you need to send less users per uh GPU card and for some like batch operations type of the processes then that total uh throughput uh total output is important. So um concurrency sometimes gets overlooked as well. together is is really um our preferred uh cloud partner uh for M3 and when M3 was shipping so their engineers just not not just like download it and then serve

22:53 it to you guys but they did some really serious optimizations to it and that’s why like Saron was saying like inferencing is actually incredible hard thing to do. there’s uh many many different layers where you can fine-tune to achieve a better like throughput or or or or the uh kind of quality of of serving it. uh things like uh there there are four four things listing here that they’ve done uh like changing the how the attention scheme so uh when a

23:23 model reads so how that uh you know information is passed between the main card and the temporary memories uh instead of passing them uh like multiplying through uh they’ve uh reworked on that KV uh block attention so that it just gets read once things like the lowlevel kernel level of the coding optimization ations uh that they’ve done uh things like offloading like non-critical image and video pre-processing type of tasks not let

23:54 them occupy the GPU but offload them with some normal like chipsets. So all of things are adding together in some numbers where you can see which which is very light in just showing it but there’s like incredible hard work to do. So they’ve managed to do all of this uh for you and serve them at twice of the throughput that even for us uh like running through our own uh inferencing uh platform. Another case study so uh as like briefly touched upon Jensen Park

24:27 use cases but I want to uh talk about it through another angle. So uh I think together was optimizing a lot on the inferencing side but on the genp park use cases there’s another side to it which is getting very very popular and critical for for this year the reinforce learning use case um with industry’s specific knowledge uh and and data then with a good foundation base model and when you finetune it post train it uh it can achieve not only the cost efficiency

25:03 but also a top result specific to the industry and task that you are running right and in the gen chance transpar cases they’ve only got like 2,000 slide deck and they’ve post trend M3 with these 2,000 slide uh tasks so here it’s a complete task of running it so so now the equation in that uh uh so the reward uh parameter in that equation that’s getting optimized is the entire uh slice

25:33 slice stack gets generated. So all the kind of like faithfulness of the text and uh the look and feel of that generation will get trend and therefore resulting in a um 4.25 average user rating for uh M3 uh slice generation and the OPUS 5 which some of you might use that on a daily basis for slice generation uh is similar level of 4.2 two three. So when you have such performance uh on par with each other and at 10 of 10% of the cost then

26:09 running these uh in production uh is really uh helping uh the case. Um what’s the takeaways? So um I think a couple points so uh minimax. So we are the foundation model lab. We uh work on uh the foundation model level and we ships the model and making sure that they’re competitive and they are in time. Um and we work on some of the architectural level uh of the challenges, some of the very uh toughest challenges there. Um and um and and we

26:44 leave it to the uh industry uh and the ecosystem to build on uh on top of that to make sure it can serve some particular industry can serve some particular geographic regions uh and working together to uh help and support our customers. The geo uh like angle could be like the European uh data sovereignty angle that gems has been working so hard on. So with together AI deployment you have a 100% European data compliant deployments running M3 models.

27:19 So you get the benefits of the cost as well as the uh data residency requirements. Uh there are also the cases where you as a customer can bring your specialized knowledge and data and get trend on compliant uh infrastructure on uh together AI and that uh result will be able to support your new business model will get you new fun raising uh fundraising and and lower the bar of entry for you to uh participate uh in that uh AI uh AI development race.

27:52 I think so. Uh with that in mind, so um we we’ll be here to for further discussions. >> Okay. [applause] >> Um one of the topics Rio kind of um rightly mentioned is the topic of RL. So um you know that’s really about taking open weight models to the next level. And so now I’m going to be joined by our VP of model shaping in Max. There you are. Can see for a sec. Who’s going to talk through um all things RL Max? Thank you for the introduction James. Hey

28:22 everyone, my name is Max. I work on model shaping and post- trainining at together AI and today I am going to give a short overview of one of our latest research projects uh the name of which is score centering uh that stabilizes of policy RL uh but before I go into any detail on that project how many people here have used uh agents for co-working for coding for all of these tasks lift your right hand how many people have trained um agents for RL for coding any

28:56 of that raise your left hand uh a few uh we should talk because we are hiring uh but um I’m going to talk about all of the challenges that come with training progressively larger agents and doing this at lower costs in the spirit of our event. First of all uh I had to include this slide essentially for uh large language models is growing to be increasingly important. These are like super like old figures by now. These are from the 01 presentation um by OpenAI

29:31 and the right one is from Deepseek R1 where it was demonstrated pretty clearly that we are able to achieve stronger and stronger results on a variety of tasks that usually require verification uh by scaling RL on top of language models. So that’s grown to be an extremely important thing in all of the LLM post training pipelines. So you would not be surprised to find that the latest technical reports all include tons of details about how uh different

30:05 organizations, different labs scaled reinforcement learning. Now the second thing that comes to that uh with that is that the price of running reinforcement learning as a stage of uh model adaptation is growing as well. Uh there have been like plenty established relationships by this point between the amount of compute you spend on training models with a rail like scaling the rollout phase and the training phase uh generating and training on more uh

30:38 observations from the environment uh and the improvement that you get from that. So as you can see you can increase that pretty far. That’s from the scale paper uh earlier this year. the results while uh leveling off at some point uh keep improving quite steadily. So that holds across quite a broad distribution of tasks. Here it’s mostly maths but the same thing uh applies to coding and many other use cases. So of course once you start thinking about the uh challenges

31:14 of trying to spend the GPU cost efficiently you start thinking about optimizing inference which of courseh we together have done quite quite a lot and because a lot of the compute here is spent on inference itself. So all of the techniques behind kernel optimizations uh model compression, KV cache compression and many many other uh optimizations they come into play here, right? But the problem is that as you start applying uh many of these

31:48 cost-saving techniques and optimization techniques for RL, you are faced with another problem. And here the problem is that uh RL especially at scale at the scales that I’ve shown to you previously is notoriously unstable. If you were to apply a very standard like textbook reinforcement learning recipe to large uh language model training setups with sufficiently sophisticated environments

32:18 uh trying to train an agent for your own organization with like a model that takes hundreds of billions of parameters, you would likely see uh your experiment fail and not in a way that uh would like be easily recoverable. That of course is very painful when you’re faced with this. We together like run uh lots of aerial experiments. So we’ve experienced that pain firsthand and um obviously as you keep needing to restart such experiments, it results in further

32:51 cost increases for you because all of the GPU compute you’ve spent up until that point is basically wasted. Well, you gain some experience but then tell it to finance. Okay. Um so how do you solve this? Let’s now get back into [clears throat] the question of uh how RL works in general and we need to talk about of policy RL. So the default policy gradient object [laughter] objective unmodified one looks like this. So essentially what

33:24 we’re trying to uh optimize the kinds of updates we’re trying to make are essentially the uh gradients of the policy of the like language model probabilities that produces the actions uh weighted by the rewards. You can add advantages here like group centered advantages you name it but the core principle remains the same. Right now the problem is that we uh in practice do not use the same model for generating rollouts and for training

33:58 for applying the weight updates. So there is the trainer part and the sampler part. What it means is that the expectation now changes. We get samplers from a different distribution. the whole uh training procedure on paper becomes invalid. And uh in practice there are many reasons why these distributions do not match. Uh for instance, you could be using different kernels at the training stage and the generator stage which is often done for efficiency reasons even

34:31 if they perform the same mathematical operations because of the numerical differences that can accumulate quite a lot. The other problem is that some operations specifically such as expert choices in mixture of expert models might also differ uh due to this numerical differences that accumulate and result in different decisions. So that also needs to be taken into account. Um and the final point is that the generator again for efficiency

35:01 reasons might be using slightly outdated versions of the weights because we want to keep our inference pipeline busier at all times without waiting for all the updates to be performed. That’s usually called a synchronous RL. So um the existing techniques to solve this usually boil down to things such as important sampling uh which is a fairly well-known technique to get back on the training distribution or things such as deterministic kernels rout

35:36 and there’s like plenty of infrastructure research and blog posts about this problem specifically but in our work uh Martin who was an intern on our team who’s like actually led that research uh uh and myself uh thought about it from a slightly different perspective. Uh can we address that uh mismatch directly at its root and that’s the core idea of the paper. We wanted to understand what makes RL unstable specifically compared

36:08 to SFT. And I’m going to speed up just a bit here. But essentially the problem is that we uh even comparing uh offline and online training uh can easily see that the behavior of uh like stable and unstable runs differs depending on the reward. So there is something not just about the nature of rewards in RL but also about the fact that the updates are online compared to the way you would run SFT against a fixed uh mismatched

36:43 distribution which would be kind of like distillation. So the idea that we proposed is called score centering and I want to stop on this uh animation for just a second to let uh it sink in. And the idea here is that uh there is um static gap between the trainer and the sampler that results from all of the differences that like accumulate due to this like numeric implementation and so on. So that gap results in a persistent difference in how uh like both

37:19 distributions behave and we can actually measure it and uh take it um into consideration when computing the updates with respect to our like actually generated rewards. So that on like in 10 20 seconds is the core idea of the paper. But the way we go through it is like this. So without again like tons of notation, the idea is as follows. Let’s have

37:49 uh this uh expected score uh with respect to the sampler explicitly written out. So that basically tells us uh how do the gradients uh behave when we weigh them with respect with the sampler distribution. Right? So if the sampler is the same as the generator the score in expectation is zero. So there should be no um difference. But now if we u do a bit of like uh second year

38:22 undergrad probability theory don’t look at all of these formulas. We end up with this decomposition. So the expected reward, the formula you’ve seen uh a bit of time ago, can be expressed as a sum of two factors. The second factor is the one we want to keep. It basically gives us the signal for the training. So that’s something that uh connects the reward and the policy scores. But the first one is something that we call the drift. And that one depends on the

38:53 average reward and on this uh expected score. So again if we are on policy that should be zero but in reality sadly it isn’t. Uh what can we do in that case is let’s just adjust the uh scores with respect to this s u uh overline that we can compute explicitly. So that’s the core idea like let’s just take uh the existing scores of uh the policy and then subtract these expected scores with

39:27 respect to the sampler distribution. In practice it means that we simply need to take the log probabilities of the sampler and reuse them when we compute the expectation. Now uh our update here becomes free of drift. Cool problem solved. But let’s actually look at the experiments. So basically there are three main results that I want to show. Number one, uh we add some artificial noise for small models like quen 3 0.6b and compare

40:00 ourselves against different important sampling techniques. The core result here is that if you keep increasing the degree of of policiness, the strongest methods that survive are the most um advanced uh important sampling techniques and score centering. But the advantage here is that we can combine score centering with important sampling because these two techniques they do not overlap in any way. So and as you combine them you get even better results. Um then as we move to more realistic settings we end up with uh

40:35 like intate quantization or some super asynchronous training which is cool if you want to save costs. And here again like the combination of square centering with important sampling or square centering standalone sometimes outperforms all of the other methods by quite a huge margin. And I do want to say that here we study super extreme settings but that uh kind of allows us to push further the uh costsaving angle. Um and then like in the most um

41:06 interesting case when we move to 30 billion models like uh quite a bit larger and train on math data sets. We’ve studied a number of different important sampling techniques as well as score centering and it turns out that uh on its own it performs uh surprisingly well. So you do not need to do any uh important corrections but at the same time it’s uh like produces this uh nice results. So even when no other techniques are able to effectively

41:38 converge, score centering gives you uh this nice boost in uh quality during training. So for implementation details, I will just quickly say that you do not need to have the whole distribution from the sampler because that would be lots of data transfer overhead. Uh you need just top 128 uh tokens and actually it’s possible to reduce even further in some cases. Uh so conclusion we propose a new perspective on [snorts] how one could think about uh trainer inference

42:11 mismatch in RL. The other thing that we study is that we propose score centering as a technique for this uh stabilization. Uh it is both competitive and compatible with existing approaches such as important sampling. Um and the paper with the code are coming very very soon. Uh thank you very much for listening and now I’m ready to give it back to James. [applause] Thank you very very very very much.