Workshop: Training Agents at the frontier of AI Industry

Read the Portuguese translation →

Author: Hugging Face · Watch on YouTube

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Workshop: Training Agents at the frontier of AI Industry · Channel: Hugging Face · 1:57:43 · Watch on YouTube

Topics: AI agents, AI evaluation and benchmarks, AI models, Local and open-source models

Summary

This workshop brings together industry practitioners to discuss training AI agents, with a focus on real business outcomes, reliable environments, evaluation, and the infrastructure needed for agentic reinforcement learning. Speakers explain how enterprise teams can use traces, simulations, post-training, and modular decision models, while stressing that reward quality and evaluation must be carefully validated. They also describe practical scaling challenges, including sandbox isolation, asynchronous training, data leakage, reward hacking, and the difficulty of maintaining models in production. The panel concludes that effective agent development depends on strong cross-functional collaboration, better tooling and education, and a clear understanding of when post-training is worth the effort.

Key points

• Enterprise customers prioritize measurable business outcomes such as latency, accuracy, cost, and risk reduction, rather than adopting a model simply because it is new.
• Useful RL environments require diverse scenarios, realistic system behavior, strong verifiers, and deliberate testing of faults such as stale data and timeouts.
• Training rewards and evaluation measurements serve different purposes: rewards can be adapted to improve learning, while evaluations need stable, reliable criteria.
• Benchmarks should be designed for a stated purpose, be gradable and maintained over time, and reflect the deployment tasks that matter.
• Agentic RL produces multi-step trajectories through interactions with environments, creating engineering challenges beyond ordinary model training.
• Scaling agentic RL requires flexible environment integrations, isolated sandboxes, reliable rewards, and often fully asynchronous inference and training.
• Reward hacking, answer leakage, broken tasks, and false positive or false negative grading can undermine training, so environments need careful vetting.
• Modular systems can assign different tasks to different models, including smaller decision models for routing and other conditional steps.
• Teams need expertise in domain problems, data, evaluation, training, infrastructure, and ML operations; the right structure varies by organization.
• Post-training can be valuable when it improves a specific use case, reduces latency or cost, or gives organizations more control over their systems.

Chapters

Start	Chapter	Summary
00:01	Workshop introduction and series recap	The host recaps a four-part series on fine-tuning, distillation, reinforcement learning, and building agent environments before introducing the industry talks.
01:37	Enterprise goals and agent hill climbing	Govind explains that enterprise post-training is driven by business metrics and must account for the whole system, not only the model.
05:29	Building RL environments from production traces	The talk describes using instrumented systems and production traces to create simulations, perturb them with faults, and verify agent outcomes.
08:43	Post-training methods and modular decision models	Govind reviews imitation, distillation, preference optimization, and RL, then discusses assigning different workflow decisions to specialized models.
14:32	Environment discovery and physical AI	The speaker discusses discoverable, validated environments, coding-agent support for hill climbing, and the prospect of agents coordinating with physical tools.
19:24	Environments as measurements	Victor Bahl distinguishes environments used to train models from those used to measure them and frames evaluation as a problem of metrology.
24:13	Training gyms versus evaluation benchmarks	The talk compares adaptable training exercises with fixed exams and explains why evaluations need stable criteria and a clear predictive purpose.
30:35	Grading, reliability, and evaluation limits	Bahl explains that agent evaluations must be gradable and reliable, while acknowledging that some real-world qualities are difficult to simulate or measure.
34:52	Stress tests and maintaining evaluations	The speaker distinguishes predictive evaluations from must-pass and red-team tests and emphasizes maintaining benchmarks as long-lived products.
40:42	Miles framework for agentic RL	Taolin introduces Miles, its training backends and model support, and explains how agent-environment interactions become training trajectories.
45:34	Scaling a Terminal Bench training run	The talk presents a large-model training setup using asynchronous inference and training, fresh sandboxes, and concurrent rollouts.
49:25	Reward correctness and environment integration	Taolin describes reward hacking and grading failures found in practice, along with fixes contributed to the OpenEnv community.
52:45	Open Env Arena and training-data validation	Xiang Li introduces Open Env Arena as infrastructure for vetting environments and reports that many submitted tasks lack usable training signals.
1:00:50	Lessons from training a 320-billion-parameter model	Li discusses unstable early learning, the need for sufficient training steps, and how older tasks can lead larger models to undesirable shortcuts.
1:07:49	Panel discussion: adoption, teams, and continuous learning	The speakers discuss barriers to training agents, the skills and tooling needed for deployment, the role of post-training, and challenges in evaluation and continuous learning.

Quotes

They really are not married to the model, but the entire system. — 03:18

In evaluation the same number is not a reward. It’s a measurement. — 23:40

whatever you have this is not all the world it’s like what we can measure — 34:19

scaling agentic RL is not just about GPU throughput. — 51:40

They’re going to care whether it works or not. — 1:42:07

Concepts and entities

post-training, reinforcement learning, reward hacking, Miles, SGLang, OpenEnv Arena, Terminal-Bench 2, Bench Flow, Victor Bahl, Govind

Full transcript (original language)

00:01 Okay. So, um basically over the last few months we we’ve been doing this series on YouTube the live streams where we we taught people how to train an agent. So, we we had this idea that okay everyone is really interested in interested in post trading at the moment, but how many people can actually train an agent that you that you use, right? That you can maybe run on your own machine with open weight models and something that you can use end to end. So, we set up four classes. The first

00:34 class was the simplest. You take traces from an improved agent and you fine tune your small model on these traces to perform a certain task. So, this is pretty great if you just want to um get a model that’s cheaper to do some kind of task that you’re doing daily and you have some traces for it. And then we we showed you how to distill like the online distillation on policy distillation. Then we showed you how to do reinforcement learning with JL. Yeah. And and finally we showed you what an RL environment and agentic environment is

01:05 and how you could build those and and what kind of decisions you would make and and how you would structure the data and the task and the runtime and all of these things. So, tonight we’re not really going to go over that content again. That’s not that’s not the topic here. We’ve got some speakers with experience from from industry that are going to walk you through their projects and their use cases. But if any of that inspires you and you think okay I want to go and try this out. This this YouTube series is probably the best place to go. It’s free and and it’s really easy to follow. We also shared all of the experiments so that you can kind of just rerun them if

01:37 you want and tinker with them to get your own experiments running. So, that’s just a nice place to go and and and try out. But um first we’re going to have our first talk. >> Uh my name is Golbin. I am a research engineer in Microsoft working with GitHub and Foundry and I’m just going to share some of the learnings from tens of customers I’ve been working with on their hill climbing journey. Um okay, so I’ll kind of cover this in four parts. First kind of grounding us on why customers would may want to do

02:10 this. Um what it is that, you know, customers actually want. And when I say customers, I’m working with like your traditional enterprises, you know, industrials, manufacturing, energy, um you know, not necessarily like the startup ecosystem. And in fact, none of them are in San Francisco. Um and then I’ll kind of cover, you know, how RL, you know, gets uh what they’re looking for, the outcome. Uh and and and then how we scale it and then what comes next. So, when I first kind of started this,

02:45 and I started this like, you know, about a year ago, I’ve been kind of doing machine learning and um you know, autonomous vehicle platform and stuff like that since uh 2020-ish. Uh and then of course, things evolved into more, you know, transformer-based models, right? And when I started, you know, I thought it was about lowering cost or customers having FOMO and wanting the latest model. Uh obviously, none of that, at least for the customers that I’m working with, is true. Uh ultimately, what they care about is they have an IT budget that maps to some strategic priority.

03:18 And then they want to improve on some business metric that actually helps them justify that spend. And usually, it actually comes down to latency, um accuracy, and de-risking, you know, as you you know, do any post-training work. Uh and it’s not just the model. They really are not married to the model, but the entire system. Um so, including, you know, the tools, you know, data, network, harness, the the full, you know, wall clock time, right? And it’s

03:50 it’s actually hard uh cuz the real world is messy. Like, yes, you can show some gains if you have, you know, good sort of techniques, and I’ll get into that in a minute. But the agents that are out there in enterprises solving sort of real workflows or some standard operating procedures, you know, they have tens of tools or some customers have 100 plus tools. And there’s a lot of um you know, issues that are outside of the model. So, what I mean by that is like

04:22 when customers care about accuracy and latency, um oftentimes we have to wear our classic sort of distributed systems um engineer hat and go deeper into what are the API calls that, you know, this agent has to make to get the outcome it wants. You know, how do we optimize all the different layers? And that involves, you know, parallelizing calls, maybe, you know, introducing caching layers, going into databases and adding materialized views

04:56 and all sorts of that stuff, right? Traditional system design. Uh and then also, obviously, with all the shenanigans going on yesterday, today, and more, uh containment is going to be a word that I think we’re going to hear more and more. There’s going to be more controls that, you know, these agents have to adhere to. Um so, this has already been going on in enterprises where, you know, they do want to ensure that sandboxes are secure. There’s, you know, least privilege, audit logs, monitoring, etc.

05:29 Uh now, coming back to sort of RL and environments, um I think they are I think they are obviously very important because without that verifiable reward signal, you can’t really hill climb. So, one of the ways I’ve been doing with customers is essentially, you know, the the most important thing is to have something out there quickly. So, what I mean by that is usually they take um you know, a an existing harness, kind of customize it, and

05:59 ideally instrument their system, uh their agentic system with, you know, good sort of hotel traces, etc. And a lot of them are already doing that. Um and then once you have that rich data with the frontier model, you can take those traces and essentially create a RL environment out of this. Uh as Ben was alluding to, you know, that I think there’s another hugging face sort of session on this and we have some skills around this as well. It’s pretty much mechanical now. Uh and once you have an environment um in there, it’s important to also perturb

06:33 it a little bit. So like inject faults because the traces that you see in production data capture, you know, some distribution but not all of it. So, you know, oftentimes I purposefully like inject stale data, introduce timeouts and then have the agents kind of practice in that simulation. Um so just just for everyone in just in case um peop- people are not familiar with RL- RLEs. So essentially the idea is think of it as a simulated world where the agent will perform its tasks.

07:08 Um instead, in this simulated world, it’ll get rewards for doing the right thing or you know, puni- getting punished for doing the wrong wrong thing. Uh one analogy I like to tell folks is like, you know, before you go and fly in an airplane you know, ideally you you know, people might want to go and practice in a flight simulator where, you know, you’ll use the tools that are provided in the simulator and ideally there’s a teacher giving you a reward um or a grade for how you did. And then you adjust accordingly. So think of it like that. So having a

07:40 simulation of the world with I usually the same APIs and um if you if you capture the diversity of data distribution, you can then have more synthetic data. That part is not the problem. The problem is capturing the diversity of data. Um and then of course making sure that you have a very strong uh verifier sort of system uh and verifiers that can and validate the system state after the agent performs actions and then ensures that things are functionally correct, and also meet your safety constraints. And then you can

08:13 score. There’s, you know, process rewards and outcome rewards. It can get pretty complicated, but the but the idea is you’re really optimizing for the KPI. For example, like I have some process rewards with the customers that I work with where they care about latency or tool call. You know, I I’ll get into the techniques, but I purposefully have rewards where I reduce the thinking budget so that it can you know, do the tool calling faster. And then you ship to production. Now,

08:43 there are a lot of methods out there. Um, I just put the math out because I do think it helps to just into it that and just kind of grok what’s going on. Uh, just a plug to Ben’s book. There are also some books out there that make it palatable for practitioners. Um, so but these are the techniques that we play with to post train. Essentially, the the output of this exercise is to get trained adapters or or weights.

09:16 And also, of course, you’re optimizing the hardness. I’ll get in I’ll get into that in a minute. But at a high level, like there’s two main sort of techniques. Um, there’s imitation and distillation. These are like your standard SFT. It’s basically you’re reducing the negative log likelihood. So, if you if you And the heuristic is like if you already have some good traces, that’s good to usually bootstrap your training with like a SFT warm-up. Um, and then if you have like any pairwise preference, you can do DPL. Some of the customers I have, they actually do collect like thumbs data,

09:47 like thumbs up, thumbs down, and they have like two two sort of you know, pair of answers. Um, there’s also, surprisingly, I’m actually finding really good um performance with just like distillation, especially like um reverse um KL distillation. So, like now this is illegal if you use like the closed models, but with the open weight models, especially like plug into I don’t know the reflection team is here but they have a really strong open weight model like beam and there’s going to be more of these large open weight

10:20 models. You can actually use those as the teacher and teach like a small let’s say like a teeny 9B or 4B model to do your task and it can like give grades. Um there’s and and and then there’s RL you know the these are your policy gradient methods just kind of quickly wrap up I think there’s you know GRPO and other techniques that clip and ensure it doesn’t diverge too much but I’ll I’ll be sure to share these slides so you can also see this. Now these are

10:52 some of the results from my post training work. As you can see here um it’s like the the ones on the right are a little I’ll get to that in a minute but if you see the one the one that’s on the left it’s not like a like a nicely sloped hill climb because there’s a little bit of art here like I tend to start with an SFT and you can see it gives really good gains basically what it is showing is like a very small in this case I use the MAI model

11:22 that hill climbed on this held out set so it generalized on the held out for some energy company it’s a very large energy company we don’t need to get into the use case but what what you notice here is after SFT I then ran some RL runs and then I tried to do self distillation which actually went down a little bit and then you know I did DPO so you can kind of see that um depending on the use case you want to play with these techniques and yeah I mean there there’s actually more

11:52 learning from just exploring um and then of course you exploit once something works or starts showing gains. Now on the right I have some fresh examples that had to deal with these decision models. I’m seeing this more and more where people are I’m sure I’m assuming most people have heard about this Jeff thing that kind of blew up last month. What I’m seeing is actually I’ll go to the next slide cuz it actually shows it better. Is if you have an agentic system, you

12:24 actually want to kind of blow it up and make it more modular. Like don’t try to use like one model for every single task. And what we’re seeing is most business workflows or most of these agentic, you know, loops are conditionals. There’s a lot of conditionals there that are great for these decision models which are cheaper, faster, and multimodal. So, this is actually for a customer that I did 2 weeks ago. And what you notice here is they were using GPT 5.5 for pretty much for all of them actually.

12:58 And then what I did was essentially for routing, you know, the good the good thing is they they had all of the system instrumented, so I had really rich hotel traces. So, then I took that and then for the initial routing decision, I basically, you know, post-trained a very small LM3 model because they didn’t want to use Chinese model for whatever reason. There’s a lot of that by the way in the industry right now that, you know, yeah, due to supply chain risk and

13:29 stuff. And then for the vision decision, you know, use a very small vision model Muse which has a vision encoder. I also tried with uh Xiaomi’s Memo 9B. It’s actually pretty good. But again, due to the Chinese model, they were like, yeah. Anyway, and then for planning, I use another reasoning model. And then you can kind of see the full loop here where I went from from there to traces to then evals. And then RL environments. And then if you notice in the middle like just

13:59 I have a really good router as well that essentially does that parado inference routing. So, it routes to the right model depending on the you know, task that’s coming. And then of course also exploit the harness optimization with Jeff Boss scale up. There’s a bunch of techniques out there that reflectively optimize your prompts and all of that, and then swapping your trainer. Um in the interest of time, I’m going to go a little fast. I am seeing a lot of need for RL environments. Uh it’s clear as software coders, like nobody’s hand stitching code going forward. All the

14:32 existing code will probably be rewritten in Rust or Go, which we will never know. Uh it’s like compiled code, but I think going forward, like we will be hand stitching rewards, and we’re going to be hand stitching these environments uh so that uh the agents can actually like learn, and there’s a lot of techniques out there uh to create realistic environments. Uh there’s tools out there like reverse engineer anything, and I’m sure there’s going to be more where you can essentially reverse engineer like uh pretty much any tool and make it realistic in your simulation. And then

15:08 and then you can um and and for GitHub, um we’re probably going to announce it, you know, next week. Um I I have some PRs out there that have that have been merged where um essentially we’re making it easy to solve the discovery problem. So, we have a standard called a genetic resource discovery. So, whether it’s you or your coding agent, if you ask, “Hey, I want a cyber gym because of all the crazy stuff that’s going on to train my models on it.” Uh if you search, you know, all the We’re actually seeing like I think there’s been there’s going to be millions of environments on GitHub, but they’ll be properly tagged,

15:41 and shout out to like the bench flow team, and like there’s finance from Hugging Face. Uh you know, we’re going to make sure that high quality ones are verified, validated, and then, you know, that you can discover. Okay, going fast. Um Now, a lot of these hill climbs, honestly, you can just do it with your coding agent. There’s a lot of skills out there that we ship, that pretty much everybody ships now. So, whether it’s the harness optimization techniques or, you know, the way to use a trainer to exploit all these techniques, um creating environments, etc. So, I would I would definitely do that. Now, I think also,

16:14 as you uh do this in the real world, you’ll realize that to achieve business outcomes, you have to extend this to the physical world. Um, that’s what I’m seeing and that’s what I’m spending a lot of my time on is to how can we uh, you know, go beyond the world of bits and into atoms and have agents orchestrate with tools uh, that could be like a tool a a robot arm could be a tool working with some instrument to verify things. And I think physical AI with robotics are definitely

16:47 going to be like the big thing because there’s something called like diminishing marginal, you know, utility in the software world right now because software has become abundant just like water. Um, you know, there’s people are going to be like, okay, am I actually achieving my outcome with all these tokens burned? And you’ll realize that the only way to achieve outcomes is to really, you know, orchestrate and coordinate with the with the physical world. Um, but I do think like there’s going to be some major bottlenecks going forward as the models

17:17 get more capable and uh, powerful. Uh, you know, alignment I I’m actually surprised regulation hasn’t kind of started. I think it will. Uh, it’s just a matter of time, but there will be controls that will force you to ship systems that are aligned not just goals but aligned to values and, you know, contained just like traditional software, you know, least privilege, time-bounded access, zero trust. There’s a bunch of techniques that we can use here. Uh, and then verification uh, as well, you know, using formal methods or

17:50 physical checks, etc. Um, uh, my CEO Satya has been tweeting about this like nonstop, but yeah, I I think it’s going to be huge. Uh, and then I think yeah, going forward, you know, we’re we’re not hands-on coding, but we’re going to be creating these um, sandbox environments to uh, you know, essentially that that that comprise of our judgment and taste and outcomes that we want to achieve and have agents hill climb on it. Uh, aligning to our values. I do you there’s a lot of despair right now with obviously the code uh, you know

18:22 traditional sort of programmers there’s despair in the math community and I think pretty much any verifiable domain is going to go through this. But I do like I like to read like Deutsch and um Popper and stuff like that and I think there are no bounds right there’s like no bounds to knowledge there’s no bounds to you know material wealth if that’s what it is but really um we just got to make sure that we stay positive and like continue to adapt and complement this powerful technology

18:52 which I think like it’s actually not uh being exploited enough in the sense like we have to steer it towards outcomes like it’s just there’s a lot of token maxing that I’m seeing but really it needs to go towards um I personally think like the physical world but also ensuring things are done safely and securely. Um yeah. That’s my talk and feel free to you know ping me on LinkedIn or socials. THANK YOU.

19:24 >> OKAY THAT WAS OUR FIRST TALK AND THAT WAS REALLY inspiring really nice way to kick it off. We’re now going to move on to Victor Bahl and Victor Bahl comes from Merkle he’s going to talk to us about environments which will be a theme throughout and take it away Vic. >> Hello. All right. Well I mean this tough act to follow beautiful slides. Um okay. So I’m Victor I’m working at Merkle we build you know we sell data

19:58 and we build a lot of environments and we build a lot of evaluations for a lot of people and before that I was at a company called Sierra and most of my life and before that I’ve been building professional agents evaluating them deploying them. But in the past few years I’ve spent most of my time building environments that are not whose job is not to to train but to train models, but but to measure them. And I think here like most of you probably build environment, host them. I

20:30 mean, uh pull them down by the dozens from GitHub, plenty of places to to to train on and and we’ve heard in the previous talk, we talk of environment as a source of of reward. Right? But I’m really coming at it from here from the perspective of measurements and I think what Govin was was saying regarding, you know, legislation, measurements, understanding, like this is really about just can we measure things? Can we trust? Can we have a good measure?

21:02 Um so we build environment to trains, we build environment for measure and and and I think these two jobs are actually more different than than they look. And I think mixing them up has been sort of the source of a lot of trouble, so it’s Open AI environment environment is the title, so you know, let’s talk environment. And I will talk about environment, benchmarks, and the metrology of AI agents, metrology being the science of measure. All right. So, I thought I thought I would sort of start start from um the beginning, which is

21:34 um Before asking what an environment is for, it’d be worth asking like what it is. And I think from from biology, um there’s a lovely answer, which is and it involves this this oak tree. And basically the same oak tree, same piece of the world, is a sort of different environment for each creature that that lives in it. You know, I mean, if you’re a human, you’re going to see timber or shade. If you’re an owl, you’re going to perch on it. If you’re an ant, you’re going to find food or burrow in it in the bark. Um

22:06 so each creature going to sense and do and and what they can sense and do decide sort of what the tree is. Um so an environment is sort of the world seen through the actor’s body. And we all work like that’s where we’ll start. And so the same for agents, like if you develop an agent in an environment, uh it could be the same enterprise, but if a chat agent, it’s you know, it can well, chat and text. Uh, you know, voice agents have different modalities, uh, different actuators. Back office agents maybe they see different things and so

22:37 all of this kind of varies and agents even though they’re the same piece of the world they sort of see different things. Uh, note that like here the environment doesn’t like it just exists. It just is, right? Um, but the moment you attach a task and a grader, now what you what you have is like something that has a purpose. It’s not just there, you know? Um, and so I think I would like to talk about reward. But I don’t want to say like this is a

23:08 whole talk. It’s actually like strictly like an environment that produces trajectories. All right? You attach a task and a grader and then you’re going to get a number. Um, you have a task trajectory, you get a score. But like we need to make a lot of choices about task, how do we grade? And and when we train this number is a reward. Okay? Um, and it’s how we judge a reward? Well, we judge it by um, what it does to the model. Like is it learnable? Uh, does it have variance at the right difficulty? Uh, is it hard to hack? Um,

23:40 it can be noisy, but it’s okay because it can be also like tailored to the model and it’s probably a good thing because you want to train your particular model like not some other model. Um, and as long as the model gets better really like the reward is kind of good. Just go for it. Um, but in evaluation the same number is not a reward. It’s a measurement. It’s not the same thing. It’s what I want to insist on. And it’s judged by what it tells you about the model. Is it reliable? Is it like So the number should be reliable, valid, and important like does it

24:13 predict sort of what’s going to happen in in deployment in in most cases. Uh, and in metrology so science of measure, we call this uh, fitness to purpose, which means the method is not like valid in general. Like a method is valid for a stated purpose, right? So, you might look a lot alike, like the same machinery, environment, everything. But it’s not the same purpose. So, we need to really think about this. Um and I I you know, the classic arc is we talk about gyms and we talk about benchmarks. That why we do that, right? I mean, the gym you go uh you want to

24:45 pick exercises exercise that makes you stronger. You can kind of do whatever you want as long as it works. Again, good to go. Benchmark originally, it’s you take a stone, you make a mark in it to measure stuff. You don’t get to change that. Once it’s there, it’s there. That’s it. You go, you measure. You can’t just have some decide to today I’m just going to make the mark somewhere else. That’s not possible, right? So, um this is sort of what I want to sort of talk about quite a bit right now. So, another analogy is like homeworks and exams. Things pretty We all familiar with that.

25:16 Uh and you know they’re made of the same stuff. They’re like exercises that you’re going to do. Um but a good teacher, you’re going to pick the homework, you’re going to for for the student as like does it help? If you’ve already mastered it, like let’s not do that anymore. It’s not it’s not important. But you can do that in an exam. Like the exam is the exam and that’s it. So, like when you build recipes for your training, um you know, totally fair game to filter by pass rate, reweight towards the right difficulty. It’s all good to craft the right reward. It’s actually like people are excellent at this.

25:50 But if you do the same thing for an eval, then what you’re doing is essentially like sampling violation. You know, so you you have to you know, the exam doesn’t care how you studied. Like you just need to be the right exam. I mean, it does care if you don’t cheat, which for benchmark sometimes problematic. But I’m not going to enter benchmark maxing and all this here cuz we don’t have time. All right. So, um based on that, like I think you know,

26:20 as people who train models and we talk about evaluation, you have to choose your evaluation. Uh I think you you know, what in real set like there are two settings, right? Real settings of course I think like KPI, real deployment super complex. I hear I’m Let me talking at slightly more granular level of like a particular evaluation. What we actually want is a measure of is a prediction, right? I mean, how will the agent do out there? Um once it’s deployed. And we can’t always measure directly before deploying. I mean, sometimes we do, but even then we also have to we also have

26:51 similar-ish problems. But something we’re building new features, something that are not out there. And so every environment in this case that evaluates sort of makes a bet about why the score on this environment is sort of going to carry on over to the real world. I mean, if you have a lot of deployment, usually you end up like checking, right? I mean, if you keep measuring and it keeps not working in the real world, I mean, the real world is not wrong. Sometimes your AB test is poorly set up. Usually your benchmark is wrong. You have to you have to evaluate it. You have to re-evaluate it. But you you do have to understand the bet. And I you you do mention Govind

27:23 that did mention realism. Super important, but I want to say something about this. So, the most tempting bet is realism. You make the environment look as much as possible as as the world as you can. But you kind of have to be a little bit careful because we don’t really define realism very often. So, like when you look for an environment, if this is the most realistic, yes, but why and how? Essentially, um because um realism like usually it does hide design choices. And if you push it to the limit, you have this sort of like what? Like

27:55 isometry, just like one-to-one map of the world, like map is a territory. You you produce everything. Well, that’s the real world. You don’t need to to do it. Um but when you think of like something like you know, in engineering, we what we talk about a model of something, it’s not what you train. It’s like let’s say the model of a wing for a plane and let’s think about a wind tunnel. Wind tunnel is you know, you want to study something, you want to measure something. And if you think about a wind tunnel, it predicts flights, but it doesn’t look like the sky fundamentally. It’s has nothing to do with it.

28:27 But what matters is the right relations between the wing shape and the force on it are there. And so, you can drop like the passenger, the the the catering, the the cloud I mean, whatever. Like, it doesn’t matter, right? What you care about is flight, and so you go into that. Of course, after you go fly your plane at some point, and so you should do that. But still, like it’s not about realism. Uh it’s more about keeping relation that matters for your question. So, the question is not just like is it realistic when we build environment? It’s more like realistic with respect to what? And so,

28:59 again, like what is it for? And by the way, like realism is not the only bet we can make. Um there are different types of benchmarks, different types of evals. Some of them make a bet on theory. You know, for example, OKGI, it’s a famous benchmark. Um but it’s based on a theory of what how knowledge works. Like, you know, it’s based on a little bit the idea that we have some core knowledge, and if we can we’re able to like achieve that, that predicts how good we’ll be at, you know, plenty of other tasks. But if the

29:31 theory is wrong, then you have a problem. Simulation. So, I’ve built a benchmark called Tower Bench, where we simulate um user agent interaction. In this case, you know, we preserve the right the right it’s more like a wind tunnel um relations, but again, here the problem is if your simulation is bad, then you have sort of like a simulacrum, like something that’s like looks like a good simulation, but really isn’t. Uh you have real work. Sometimes you’re going to have environment that tell you that are built on real things, like real actually like identity. Like, this is what people did, or like real traces. Um

30:05 but then here the problem might be like, yeah, it might be skewed or stale or how do you get there or contaminated from the model already know about it, etc. etc. And you have also real users, and even in the world of benchmark you do have places like arenas in which I actually reproduce AB testing. Here the problem is you don’t have replay, so I mean, you do can test multiple users, but you can’t really train on it. Um and of course, the failing is that the users or you might not be your users. Um that’s problematic. Uh I’m not going to talk about this

30:35 actually, because it’s about like we don’t we evaluate agents and not models. Um I just at least mentioned that like models are made of many many many I just mean of many many more things than just a model. Uh at Sierra where we’re building just very complex models agents or I should say for for customer support most enterprises agents I mean it’s really an MLops problem essentially. It’s just like it’s a lot of like it’s very very complex. It’s not like one model inside like a tiny loop

31:06 harness. Like that’s not at all what it looks like. Uh I think you you were like spot on. This is exactly what it looks like in enterprise. So all I wanted to say here is like essentially you have to think about evaluating an agent. And in a case in in many cases we don’t really care about the evaluator doesn’t really care what’s in the box. Right? But you know, the box does matter. You should like you bring your box and you should also report like the cost the late input latency, but all those things are also like very important. But after we have an evaluation of course like there are questions around

31:37 uh what are we evaluating? Um all right and I’m going to be on time. Perfect. Another being one uh and that’s something I think like is really understated. I mean I we know that, but I think it’s easy to forget. Um a measurement can a measurement has to be something we can grade. I mean it has to have a clear kind of right or wrong or some way to to to be judged. Um

32:09 or at least one outcome people agree on. And I mean when I built Tower Bench for user the customer support or you know, user model interactions I mean well it has to be graded fairly for it to be a good task. So you know, uh we have to have a reliable user simulator like consistent grading which means we did like verifiable rewards for example as opposed to like element judge. Um it has to be no ambiguity in the rubric you’re using. You You be able control to say to control say like yeah, especially when I have user simulator that no the agent is making the mistakes

32:42 not the user simulator because otherwise what are you really measuring? When you have LM judges, I mean yeah, you make sure the verifier is actually like not at fault in most of the cases. So you have to calibrate. But in general like that restraint that that really can restrain like constraint type of task, you can actually build that are actually going to work. Um So in reality most of the I mean if life only were like this, right? It would be so nice. But it really isn’t. Like most cases you don’t really know what’s good or bad. I mean it’s not that simple. Um Like for example in dialogue bench we

33:15 don’t we didn’t do anything about um cuz this de-escalating uh customer an emotional customer because we don’t really know how to simulate the emotional pattern of a user and how to get I it’d be great but I mean the the truth is that it’d be probably not very good. Like we don’t really know what we would be measuring. So you’ll see environments where people tell you it’s there. But is it? I mean or is it just like just a bad simulation or something that is like full of errors because So uh you have to be very clear about like how the quality of the task I mean

33:46 there’s some tasks that are just like not good. And you kind of have to live with that because usually the the thing to remember is like reliability bounds validity. If it’s not reliable, it’s just not valid. Like you can’t make anything valid that’s not reliable when you measure. That’s it. So when I build benchmarks sometimes I do think they’re cleaner than the real world. But it’s not really a failure of realism I would say. It’s more just like the price of measurement. I mean there is a price to pay to just be able to measure something which is that we have to be something measurable. It’s not always

34:19 easy. So it’s why I love this stuff like streetlight effect, right? You always remember that whatever you have this is not all the world it’s like what we can measure and we kind of have to look under the streetlight. Like we you can want to go look somewhere else but you you just can’t. Or you build like better technique better benchmarks. So this is about like evolving the measurement tools themselves. And that’s like a whole different problem, right? How do you build different benchmarks that are which I’m doing We’re doing a lot of that at Merkle, and I think it’s fascinating problems, but you have to think about like what’s your instrument.

34:52 Um all right. The one thing I think I’ll stop with uh is the average is not the only thing you care about uh in industry, not at all. Uh when you build anything in the real world, like you have to stress test. You keep regression suites. Uh there are must-pass tests for things that are really you don’t want to see at all. Um I won’t tell you how many times we just had a regression test for like the thing we knew to

35:23 not at Cera, I must say, but at other companies. You know these things should not happen for different reasons. Um so you keep that in your regression test. But this isn’t sloppy sampling, right? It is just you didn’t measure a different quality. Like if you want a predictive eval, you’re making frequency claims. Like it’s going to happen at this frequency. If you If it’s a must-pass suite, this is about conformity. Like it has to conform to that. I have a PRD for my product that it has to happen, otherwise I’m in legal trouble, for example.

35:53 Uh and then you have red teaming, which we do a lot because we understand like especially for security, you have to do red teaming, but this is a purely existence claim. You don’t care about like is this one counterexample is enough, but then you have to associate a cost to it. Um so here is not about the quantitative. It’s just about existing and being bad often. So again, like this is another type of evaluation. This is a very important one in practice, and especially now when we have problems with security, like it’s not about average. And one thing I’ll say is that this ties to when we People ask me like what my

36:25 how my agent should be doing on something like Tower Bench or some other benchmark. Well, I’ll say like you have to be kind of careful because in many ways the benchmark is easier than the real world. Uh it doesn’t have any ambiguity. Uh it’s simulated user that are often far simpler. But Um, in many cases it’s it’s it’s so you should ace it, but it’s also a lot harder than production data in many cases. I mean, you know, it has a lot zip flaw distribution where often it’s like very simple cases like you just want to refund, okay, like two tool calls, you’re done. Then you have like these really hard cases and so as a

36:59 designer of an agent, you should think about like um, and because of course, sorry, as a benchmark designer, you want to have two hats. You want to both show new capabilities. Uh, you want to you want to show but you wouldn’t do you do want to keep the model fail somewhere, right? Does that make sense? Um, but so you should know your traffic. Okay, I think this is like super important. Understand what’s going on. Uh, map on to the benchmark. If you use a benchmark, make sure you map on to different type of task or re-weight, understand what really matters for you, uh, and remember that the real arbiter is going to be deployment at the end of

37:32 the day. Um, so I think I’m out of time. Uh, I’ve one I’ll I have more things that I can share on other slides. Um, but yeah, I’ll I’ll say I’ll say that cuz I think that’s important. You’re going to live with your evaluation for a while. So, a good evaluation is one that you maintain. I cannot insist enough on that. Uh, because often like we start deployment or you pull an eval and you’re like, “Oh, it’s fine. Like I’ll just do it for 5 minutes.” And for training, it could kind of be good because you might train

38:05 your model and just better and you move, but that for your evaluation doesn’t really work like that cuz you’re going to want to live with it for like a long time. So, it’s a maintenance problem. Uh, when I’ve, you know, for me, I’ve really seen that when we build like again, our benchmarks, like we keep like it’s not the first time that it’s good. It’s just like you keep having to make it better, new versions, cleaning up. You find that it’s better. So, you have to treat like a product, honestly. That’s what I would I would recommend. Which is a little bit different sometimes than like your your your training data which because once your model has learned it to some extent, maybe you you could say, “Okay, like let’s move on, right?”

38:38 Um, all right. So, maybe what what I want to say is like whenever you do something and you want to measure how your agent is doing and you already chose this number. Like I would really think about like is my number in my environment particular like is it a reward or is it a measurement? Like what do I want here? Uh if it’s realistic realistic with respect to what and in general like what is the bet that this environment is making in terms of predictive predictive power and I would say like like Govind said like in

39:09 enterprise the bet is like the is simple like if I do something in the enterprise my internal benchmark and you do AB test and it’s well done and it fails my benchmark is wrong. You know what I mean? That’s just going to happen and that’s the everybody has experienced that like very very many times where you think you cracked it and then you go it’s like no it doesn’t work. So that’s something that like we all you know we all experience. And then what are we measuring? Like the model model plus harness like etc. etc. like that’s very important. Can it be

39:39 graded? You won’t believe how many time it’s just you have many tasks it looks very rich but it just degrades something anything because the task are maybe very fancy but fundamentally like they’re just not gradable they’re not fair. And who maintains it? I mean you if you are the one building it or someone else but you’re going to have to live with it. So you should think about like So in general never good I mean never good in general like just never it’s a good benchmark. You know I know we’re going to tag benchmarks are really good but good for what? For what purpose? I think this is

40:10 something I would like people to really just think about because in industry like we really have to think about that. And thank you very much that will BE IT. REALLY GOOD TALK. REALLY GOOD. >> OKAY SO THAT’S OUR second talk. That was really cool. Um so much kind of industry insights there. Stuff that we’re kind of not getting anywhere else. Um next we have Taolin from Radix Arc. Tao maintains Miles an amazing framework for training so something that you can take away and use. Um if you come on.

40:42 >> Cool. Hi everyone. I’m Taolin. I’m engineer. up. >> You got it. Okay. >> Yeah >> you’re good. You’re good. >> Yep. Hi everyone. I’m Taolin. I’m engineer from Radis Arc. Uh today I’m going to talk about how to train your agents with Miles. So, first of all, what’s Miles? Uh it’s a open-source framework. Uh it’s developed by Radis Arc. You probably know or don’t know that it’s also the team behind SG Land. So, obviously, we use SG Land as the inference engine. And

41:16 but we support a lot of training backends, including Megatron, FSDP, and Torch Titan. Uh we supported basically all major open-source uh models. Oh, and for many of them, for the our most popular ones, we even have day-zero support, like the models mentioned here. For example, Meow Spark Open and Nemo DeepSea Q&A models. Uh so, day-zero means you can start your post-training on the day that the those providers

41:49 release their weights. And Miles can scale from a single GPU to hundreds of them or even more. And it supports both Nvidia and AMD GPUs. And like there, we have a lot of collaborators in all the stack, including chip companies, clouds, um Frontier Labs, or yeah, a lot. Yeah. So, now let’s talk about uh

42:19 like agentic RL. So, you probably know from Ben’s lecture, but well, or from elsewhere, that like, for example, in a traditional RL is uh it’s very simple. You have you generate a response, calculate the reward, then update the model, and then repeat this again and again. Uh but for the agentic RL, the difference is uh the generation part is

42:51 no longer a single response, but it’s a multi-step interaction. So, the agent will decide what to do next. Then, the action is sent to environment. The environment sends the feedback back to the agent and do this back and forth. So, we call this full interaction like a chat history. We call this a trajectory. And we call those uh and the reward. And we also call the And we also have the reward. So, this

43:24 trajectory together with the reward becomes one training example. Now, we can see how we connect How How we can connect environments to Miles. Here, I use Open AI as an example. Uh so, for each in with Open AI, each environment is like a mini server. It exposes APIs like reset and step. Then, all you need is to write a agent

43:56 function, which is a short loop. Before the loop, it first, of course, resets the environment, but then it calls the model, takes the action through the step, and do this over and over again. After it finishes, it will also get the rewards or get the score from the environment. So, after this agent function, you can get a trajectory and a reward. So, those are the things that you will send to the

44:27 training part, use them for training. So, as you can see in this setup, uh your your existing environments can be connected to the training without rewriting the entire training system. But you may also say that my use case is more complex than that. For example, I don’t just want a single agent loop. I want to train with a specific harness. For example, I want to train with cloud

44:59 code or open code. Or in other cases, you even want a harness inside your sandbox. So, it’s more like how they actually work work in the real world. So, that they are those different use cases are all covered, but obviously they cannot fit in the same interface. Uh that’s why in Mouse, we figure out it’s better to provide a integration point at different levels. So, so that you can connect your harness

45:34 and environment in the way that’s most that most naturally fit. Now, let’s see a real training example. So, here we we are using Mouse to train to do post training on GeoM 52. This is a large MOE model with more than 700 billion parameters. And we use it to training Terminal Bench 2. So, Terminal Bench 2 is a popular post popular agentic coding

46:06 benchmark. So, before going to the setups, here are like two important design decisions. First, we are using something called fully async RL. What is mean it that means is inference and the training are running on separate groups of GPUs. So, they can work at the same time. This is very important or it’s at least a very popular choice for agentic RL

46:40 because for agentic RL, some some episodes or some environments as you will say will take much longer time the others. So, you cannot make if they are running starting from the same point, uh the slow episodes may leave uh training GPUs idle. So, we have to use fully async to make sure that the training GPUs can be always kept busy. The second

47:11 choice here uh the second design decision here is we use a fresh sandbox per episode instead of uh reusing the same VM. By doing this, we are basically dedicating the complex logic of doing isolations and scalings to the uh to the actual sandbox infrastructure uh so that like each layer can do the part they are most good at. We support uh many uh

47:46 cloud sandbox providers and also open-source sandbox providers like Agent A.I. So, in this setup, we will use 64 GPUs. Uh that’s 64 B3s B300 GPUs. Uh and 32 of them are used uh for inference. 32 of them are used for training. As as we just said about the fully async, so they we need a different set for inference and training. And then, in each training step, we will

48:22 have use eight uh tasks. And for each task, we will run them for eight times. So, overall, it’s 64 trajectories for each step. But, we will use 128 sandbox running concurrently. So, as you can see, the number is different. 64 trajectories, but for each step, but 128 running at the same time for inference. That’s because we need to match the

48:55 speed between inference and training. We want to keep the rollout pipeline busy, but in the same time, we want to make sure the trainer also have enough data to work with. Oops. Here is the result. So, for each step, it will take roughly 4 to 5 minutes. And after 100 steps, we can see the reward goes from 0.44 to 0.56.

49:25 So, uh even this is a like real relatively short run, we can see a very measurable performance improvement. But, like that’s not the most interesting part. The most interesting part is all the way all the things we found out through all those experiments. Uh first of all, it’s about reward correctness. So, for RL, as you can see, the models are

49:56 are rewarded to get uh to try to get higher more rewards as possible. But, if there’s a shortcut that the model or the agent can earn a reward without actually solving the task, they will try to they may discover it. Uh they may try to use that loophole. We call this reward hacking. So, for one example, uh for for example, in Terminal Bench, uh Terminal Bench provides a reference answer inside their data set. But, if you put that data set

50:32 in the agent sandbox, the agent might find it out and just use that reference solution. And the second example is more trickier. Uh the agent may be able to modify the grading logic. So, the grader can uh report success, but actually the task are is was not completed correctly. All those cases are false positives because the model doesn’t uh are the model gets reward, but it actually

51:07 shouldn’t. But actually actually there is the the there’s also problem the other way. That’s false negative. Sometimes the model may does the right thing. It may have already solved the task, but the environment report a failure. For example, it might because a greater bug or it might because the uh the infrastructure itself was flaky. All the cases I mentioned here are actually real issues uh we found uh in

51:40 Open Eing, but we fixed them and upstream these changes uh to Open Eing, so all those fixes can benefit the whole community. And here I also want to thank Ben and the Hugging Face team for reviewing and helping landing those diffs or those PRs. Um so that’s almost the end to my talk. So if there’s one thing to take away, I think is uh scaling agentic RL is not just about GPU throughput. It’s also about flexible

52:15 environment integrations, strong isolations, reliable rewards. And that’s what we are what we’ve been working on with Miles. That’s it for my talk. If you haven’t tried Miles, please give Miles a try. If you have a question, you can contact me. That’s it. Thank you. >> Okay, thanks Tao for that talk. That was really great, really practical things that you can take away and apply with

52:45 Miles. The next talk uh up is by Xiang Li from Bench Flow. And Xiang Li’s going to talk about a project that’s running right now, Open in Mario, which you can go and try out straight away, which is great to take away. >> Hi everyone, Xiang Li, I’m the founder of Bench Flow. And um so previously uh we already seen Ben talk about uh how to do GRPO. We’ve heard from Govind that was the motivation to do RL in other places. It’s like the reason why they do that. We heard from Victor how to do emails

53:18 perspective, like how to make sure emails fit to the purpose. And we also heard a great talk from Tao how to set up the infra to run RL. And so, today I want to talk about from a training data perspective, how do you find the best training data for your RL? Which is also why we built Open Ended Arena. And and thanks basically this thing exists because of like Ben’s organization and also compute by from our friends in NABIUS. So, let me get started. So, so, Benchflow we are a frontier environment lab for AI agents. This year

53:53 we made a few very popular benchmark skills bench, cost bench, this post-training arena, Open Ended Arena, and frontier physics. And um the Open Ended Arena is basically two things. One is a data infrastructure infrastructure to like a as a test bed to vet any training data in any training data. It is also going to be a data set with the the most validated environments. And this built on Open Ended and we are like live with Open Ended Arena.

54:24 So, before we get started, let me do a quick survey. Raise your hand if you ever trained a model. Okay, so almost every every one of you. Keep your hands up if you train models bigger than 10 billion in parameters. A lot of you guys. Okay, lower. What about how many of you have trained models larger than 100 billion in parameters? Raise your hand. Okay, two, three, four people. How many of you have trained models larger than a trillion parameters? Any any hands? Not even one?

54:56 Okay. So, in our literature review, there isn’t a research that post-trained or basically in the last year or so, there are a lot of research on how do you make the environments how do you make environment how do you scale the training data etc etc and the test bed for these models are usually just a 9 billion models parameters 32 billion parameters no open research has larger than 122 billion model parameter and there are a few open ones but on these are all like big labs

55:30 like GLM cursor and deep seek so there hasn’t been a very academic or very uh uh like no no lab released versions previously we trained models on the 9B parameter model size on the 32 model parameter size for the data for the model we use on post arena it’s a 27B but so today the thing we’re going to do is introduce you with the open arena is and also what we learned going from training a 32 billion model parameter to training

56:00 something like GLM 5.3 flash which is a 320 billion parameters model and before I get started cuz I have absolutely no idea how what what it what it takes to train this model so I asked a bunch of people so someone said count the e box not the steps basically you you want to see given a batch size of let’s say 16 you want to see at least 100 steps I didn’t know about that for a large model and um there’s a data company told me that I’m check at past at eight like we don’t want task that I had

56:34 no no no pass to be in the training thing to be to be in the training and um also be aware of reward hacking you know you have to find the right harness for the base model you want to train like for example in GLM 5.3 using mini suite agent and cloud code can have a over 14% in in accuracy and also the most important one which I think which a good feature in open event with post arena open arena does is to vet the environments not just like the trainability but also reward hacking

57:07 false positive false negative cuz in Evals, a noisy verifier may not be that important if it’s like again if it’s reward hacking prone. But if it’s in training, just 1% of reward hacking can just like teach the model very very inclined to take the short shortcut. So, Okay, so this is how we score a RL environment or training data. So, basically we want to think the post reward as a function open reward as a function about given any model.

57:38 Like in our in our current competition, it’s a 27 27B model. We want to fix the Eval set. Oh, so you’re going to provide the training data. You’re going to give, you know, 50 to 100, maybe even more environments that improves the Open Eval that improves the models on the Open Eval on several domains. So, right now we have a strong focus on cybersecurity. So, we are help climate cyber gym. We’ll give you a QR code to scan to how to participate the competition.

58:08 And um we’ll fix the method basically GRPO up to 4 hours. And um we’ll evaluate on 40 private tasks. So, um There are So, we in the last like 2 weeks we done like 40 tasks on eight domains. But um for the next 2 weeks we’re going to do a focus on cyber. And um basically we were going to rank how good your training data is on the help line. So, kind of like this. So, in the over the last 2 weeks we’ve got over 80 submissions from 35 teams. And um basically with the same model we

58:41 found that uh over the 170 trainings, like 125 ended up being like untrainable. So, basically most environments are not really trainable or like a lower performance than baseline. And um on only like uh only like 50 50 environments that actually have the signal. So, which means even the environments we submit to participate hackathon are broken. And um in addition to this, so when the the moment we start announce this competition when we have the access to

59:14 the compute. We basically scraped all the all the possible findable our environments out there. There’s about 90,000. And we basically label So, this is the part where we use the jailbreak 5.3. Basically labeled all of them and um only like 8,000 are like have trainable signals for jailbreak 5.3 flash. And um only like 1,000 of them are actually teachable and we filter out ones with like reward hacking or uh false positive false negative. There’s only like 800-ish tasks left. And yes,

59:46 this includes the MIMO tasks that are open sourced. So, basically this is a list of uh failure modes about environments. So, sometimes the So, the most of the time like the agents just able to solve the task and a lot of tasks are like just broken. Basically, the task is just broken, can’t run. And uh there’s a lot of task leakage, like answer leakage, and this has been such a big problem in something like SweepBench for for for a long time. And yeah, basically the

1:00:17 uh the from each like if you have you also have to pass at least one per eight trials. So, that is basically drastically reduce the total number of trainable tasks. And um we sort of want help OpenEnv become a become a place where after vetting all the environments with OpenEnv Arena with with the submissions like the which is new environments or new synthetic environments submitted by participants and existing environments. So, yeah. And um so, a little intro a little bit

1:00:50 introduction about what this is, where we are, and um and also there’s a few things. So, we’re going to talk about the few things that we learned about post-training on a 320 billion models. So, uh so, like in the first uh one one week and a half when training the jailbreak 5.3 flash model, and uh basically I gave it 260 tasks from the 800 tasks that we I selected. and I just basically the learning signal is like not very stable. So I trained on like 12 steps like over like like five rounds. Basically the thing the thing that I see

1:01:25 is the benchmark that I wanted to help climb it doesn’t climb but it like kind of climbs on on other benchmarks. So which which was weird and then I realized okay I’m not training on enough epochs which I think I just had a conversation with Tao. He’s like okay yeah don’t don’t count anything that’s less than a a hundred steps. So there’s that. And another thing I found is that so in in initial in in in beginning I downloaded a lot of like earlier academic task like open sweep T-Max like these tasks worked really

1:01:58 well on small model like 32 billion models in the papers but on actually these like 30% of these tasks taught the larger model to use less tokens less two calls. So uh basically this is a common sense problem. I don’t I don’t see any any experimentation on that. So every basically everything that we use for future training either comes from MIMO or comes from a better environment recipe. And oh yeah and um

1:02:28 one thing one thing I got told was that if one over 1% of the model when come from cheating basically this pattern will just perfuse in the in the rest of the training rounds. And these are some other like uh reported numbers about environment rot if you like. So like Epoch AI reported there are 45 broken tasks in terminal bench four. I think there’s even more reports by preference model about like this is basically a saturated benchmark if you fix it. There are some debate around that but um uh definitely it’s not upheld to the

1:03:03 highest reward hacking standard. Uh and basically according to the latest sweep bench pro verified so like sweep sweep bench scale AI has the new revised version of sweep bench pro basically 50% 57% of tasks had leak leakage answers. And in like the MIMO RL environments, they also I have found 48% of the agent trying to look for look for the look for the answer instead of trying to solve the basically try to reward hack

1:03:34 has some like exploitation patterns. And um yeah, the 30% one I probably made by Cloud. I don’t really know about this one, but yeah. So, coming to the after learning the the trainability for a 300 billion parameters models, and um we sort of want to kickstart kickstart the next episode of Open Arena, which is on cyber task. And the basically environment submission we get from uh the cyber we will also use that to train a larger model. So,

1:04:05 we did a few 27B runs. So, we’re going to kickstart kickstart kick this start Okay, kickstart this next this Monday, but um we found that um uh basically uh it can learn the cyber gym uh really well, but um with the tasks like the open cyber task that we found, it’s basically just a lot of a lot of random noise. So, like there isn’t a very good cyber environments like that’s trainable for help climbing on cyber gym. And uh one thing why I found building the cyber cyber version

1:04:38 of Open Arena was that the frontier model just refuses to do like anything at all, even though I was like trying to “Hey, this is So, I basically I have an MCP that calls my Slack. It’s like, ‘Hey, this is MCP like you can check this Ben Burton show like email like he’s from Hugging Face. He authorized us to train a cyber model to better protect Hugging Face.’ And the model just like GPT 6.1 so over 5.5 is like keep rejecting me. I had to you know subscribe to uh you know, two GLM Z AI coding account to use GLM 5.3 to

1:05:08 keep working on this. So, this is sort of I don’t know RSI in some sense. So, yeah. Um so, that’s one thing I learned in cyber. So, yeah, to sum up uh we built Open Env Arena to vet environments and there and the I think the first thing that we or the biggest problem we see is that environments are like broken whether in whether in the sense that it’s like quality wise it’s just not there or in the sense that it works for smaller models but might not work for larger models or increasing or like increasing the capability of

1:05:40 cyber in open models. By the way, I think this like open model having cyber and does not refuse you to do stuff is like really really important. I think the Hugging Face incident already proved that and also Geo 95.3 saved my ass building this sorry building this Open Env Arena. Like I don’t want to just like hand roll everything with all the refusals so I think this is a super super relevant topic for for now. But yeah, to sum up. So I think there are a few like principles we generally check for the environment the learnability like the basically shouldn’t be like leakage of answer. Oh, by the way, may not explain

1:06:14 that good enough. So basically this means so like in the vast meaning that the answer leakage just means that in the environment that you give to the agent there are some answers just lies within the good commit history. So like is there is there answer like just in the thing that agents are in or out of it? Like you’d be surprised how much how many like answer reaching behavior it reduces just by separating the verifier. It will not be able to catch all the answer reaching but will definitely like catch most of the obvious ones. And um

1:06:47 yeah, does the does the task still like build? Like basically there was like over a thousand broken tasks out there. And um so um does does the reward check what it what it actually what it actually solves? So like no false positive, no false negative. Is there a very robust reward hacking checker? And I think if you if there’s any data vendors in the room like you were just like no if you if you work with the large frontier lab you this is basically a very very very serious thing that should be solved.

1:07:19 And uh yeah, most of all like the whole purpose of bunch uh Open Env Arena is to see if there’s a help climb training on this data. It’s like similar to like I sent you like a sample of 50. Can I already use that 50 tasks to do some help climbing? Something like that. So, yeah. Uh so, that’s it. Uh join the competition using the QR code on the left. And if you want to connect uh or you have you want to help you want me to help you build your in in-house emails or data or help you train models,

1:07:49 this is a QR code on the right. So, yeah. Thank you. >> That’s a really nice talk. Really good talk. Well done, man. >> Yeah. >> Cool. So, each go around and just ask each other a question. So, just have that in the back of your mind. Uh like the first kind of question that came to you about each other’s talks. Uh so, the first question that I think is really important to Hugging Face and like kind of most people that are trained like like something that we learn is Yiannis’ talk is that most people haven’t trained a 100 billion parameter model, right? So, um like what is it that affects the kind of

1:08:24 long tail of models? Like like what is the friction that is is tripping up most people when they when they um start to take a they have a task that they can see that a high-coo or a uh some kind of nano model that they’re using in an API is very capable of fulfilling. They’ve done the napkin math of uh inference on AWS and they say, “All right. So, I can just get a 27 million parameter model and I know that it can do this. It just needs a bit of SFT, a bit of RL, and I can do it.” What is it that’s tripping them up uh and and stopping that going into

1:08:56 production and leaving them running on on high-coo and and kind of paying this extra cost and these kind of things? >> Yeah. >> Or you >> Okay. I can start. So, yeah, one thing when building both for cyber and both for just vetting all the open environments is I don’t think there’s a very good collection of open environments that you should like just kick you start for training a model. Like even though the MIMO tasks are broken here and there. So, I think there’s a real chance to uh to build or like to to exist. Like we

1:09:26 definitely want to help build a collection of audited, vetted environments that can use for for post-training cuz um due to the larger effort in the community, like the recipe for post-training is largely commoditized. I think there’s there I think it just there is a tutorial somewhere to hey, you can post-train miles on Chimera K3, which is a 2.8 billion trillion parameter model. But um the the cook cookbook usually is hey, overfit on this benchmark and then that there’s it. So I think the better like

1:09:56 vetted set of environments that would definitely help. >> Okay. As a just purely as an example to begin with before you even move into that use case. So we can get to >> Example would be so like if MemGPT did not release the 7,000 environment they used, it would be really hard for me to come up with the set of environments to for like just to improve coding for GeoLine 5.3 flash. >> Mhm. Okay. Yeah. Anyone want to go next? Let’s go to Victor. >> Okay. Um I guess I’ll go so there’s actually a lot of model fatigue for the customers that I’m talking to. Um

1:10:28 maybe I’m I don’t maybe I’m not talking to the SF sort of crowd. They really are under a lot of stress because their budgets have been stretched thanks to the coding agents, which they will continue to kind of accommodate, but at the same time you know, we’ve been in this cycle for I don’t know, 2 years and you hear this in the news, like where are the outcomes? I think that’s top of mind for the people that I’m talking to where

1:10:59 you know, they have some budget carved out to support uh you know, some other business function and the metrics that really matter, which is like at the end of the day it’s increasing revenue or reducing costs, it maps to kind of like that, right? There’s actually um yeah, not that much of a productivity boost. >> There is productivity boost for coding, of course. There’s I think all legacy code is going to be rewritten. That’s already happening. That’s going to accelerate now

1:11:31 given all the stuff that’s going on with the vulnerabilities, right? And all that code that will be rewritten, I think we talked about earlier in Rust or Go. No human will ever understand that. Going forward, any change to software system will be introduced via an agent. I I think that is for sure found a PMF. But, code is always a means to an end. And I think to your point about Haiku, which I you’re probably asking the assumption is that it’s for coding, right? Or is it? >> is that the task is like very feasible.

1:12:04 You know, I I I think everyone that’s looked at open-weight models has like identified certain like this for example for like projects that I maintain, I have a number of like auto filters on pull requests that will just group them up. And and I know that I could fine-tune a model and and it would perform that, but I have never done it. And then And then and that what are those challenges? >> Yeah, I I think after coding, the one that I’m finding really good PMF is a product market fit is ops. So, any ops. I think that falls in ops, there’s obviously SRE agent SRE related ops, there’s data ops. Um

1:12:39 For sure, that’s where actually some of my hill climbing experiments have been with customers. Uh Why did you bring up Haiku? I’m just curious. >> Um I I was bringing up like the smaller tier of proprietary models where I personally see like the least value um because you can put open models on cheaper inference and it seems like that’s the area where >> Well, I’m assuming everybody is jamming up there you know, architecture, right? Uh Right? And

1:13:10 because of small models like uh that If you in all these ops, right? It isn’t a chat experience. It’s like a bunch of decision boxes that have a restricted option set that you need to distribute probabilities over. So, that that that’s actually I think going to we’re actually making a huge bet that this holiday season, like last holiday season was all about coding, this holiday season is all about making your system more modular and introducing decision models. So, anyway, I’ll stop there.

1:13:40 >> Pretty good answer, yeah. Any other answers? Do you want to come in? >> So, just want to make sure I understand the question. So, it’s about using small models. Is that And >> Yeah, I mean, if I was to simplify, I’d say like, what what is stopping more people from training their own agents? >> Yeah, yeah, yeah. Um I mean, the the artist training their own agents like and lots of can go into that. Like, the reality is that um the reason companies like Merkle or Delta exist is that it’s still

1:14:14 quite costly and complicated to create good data. Um part of it is need to be maintained, needs to be built, needs like it’s just it’s just it’s just a hustle and it’s just complicated. Um So, I think you know, that that’s one obviously like limitation. The other thing I think that I’ve seen, which is more very practically, you can train your own model, um companies don’t know how to do ML ops anymore. Like, they don’t know. Like, it’s just it’s a real hassle. I mean, I for me, I’ve seen that uh where

1:14:46 it seems to me what you when we see for example your diagram, right? I mean, for me, when you think about models, when you think about industry, you start your career like, oh, I’m going to deploy a model. Super cool, I’m going to train a model. Then you realize, okay, well, I mean, really training a model is really like not the problem. The problem is maintaining it at the right level. You know, like, okay, how do I do that? And you’re like, well, actually, I don’t need one model. I need a bunch of them because actually, I don’t, you know. So, you realize that just like we have we we build cognitive architectures, like you

1:15:19 build architectures really. You just don’t build a model. Like, nothing is a model. It is just always an architecture with many different pieces that you have to maintain. Each thing needs to fit together. There’s a skill for it. It’s called ML ops. I think it’s going to come back if you want to be able to do that. But I really have felt that the moment like large language model came about, there was this idea of like, well, ML ops is there. You don’t need to care about it anymore. Just be put a big model. It’ll solve everything. You don’t have to maintain. Done. Now, if you want to go back to small model that you train yourself, you have

1:15:50 to go back to this. That needs to change completely the way you think about your your system. Cuz now you have to measure everything. You have to do like canary deploys. You have to be able to do all sorts of weird thing that people don’t do at all anymore to maintain their models. Their whole stack is not like this. Their models are not deployed by people that have ML training. So, they don’t they think like My biggest My biggest space people is that people file bug reports. When you do ML, you don’t file bug reports. Like if the ML is working at 80% accuracy and it’s 80% accurate,

1:16:21 don’t get Don’t tell me there’s a bug. Like there is no bug. It’s just doing what it’s supposed to be doing, you know? But people don’t It’s not the right mention like you have CI/CD. You have like DevOps. But it’s not the same as ML ops. Like it’s not the same problem. So, for me, biggest I think friction point that you have to go back to a very different way of thinking about ML and to think about statistical models in your stack. And I don’t think people want to pay the price for that or at least they don’t have the right tooling. And so, I’m I would actually bet that there is going to be a lot of tooling around around this being built. And I think that’s going to be the biggest change is that,

1:16:53 you know, we’re going to have to see just like the new migration from DevOps CI/CD to like just basically ML ops became the the norm because you’re going to have to unstuck yourself from big models because they’re going to, you know, you you don’t want to have like a single provider. You want to own your data, etc. But we need tooling. We need tooling and I think we are transitioning also skill set, people learning how to do that. >> Yeah. And I do think I do think as an industry we’re tackling that, right? Like I mean, tooling around post training is is going crazy at the moment from us, from Hugging Face, and yeah, from Prime Intellect, from from Thinking Machines. Like there’s a new training

1:17:26 framework post training framework coming out every week, it seems. Um so, yeah, it is getting easier. >> But but when you Oh, sorry. >> Go ahead. >> But when you use words like trajectories and stuff, you lose audience because a lot of these folks do what Victor was saying. Um you know, people are not aware and it’s hard to keep up and I think we got to productize it and have a value prop like this is for continual learning or this is for something increasing accuracy, reducing latency. >> Yeah. I mean, I was working like uh like in a solutions org in like 2022 when

1:17:58 ChatGPT was first released and and I noticed like a swathe of engineers just like pivot their careers out of ML MLE, right? And I think that’s exactly what you’re speaking to. Like just a load of engineers just went like, “All right, we’ll put all those weights down now and we’ll start prompting.” And and they just grew another set of skills for a while. And then yeah, it’s an education problem that we kind of need to get back to. Yeah, that’s a really good point. Uh Tao, did you want to add something? >> Yeah. I think I agree with Victor that one of the larger challenge is that there’s a huge friction inside in the

1:18:29 entire RL stack. So first of all, you need to source your GPUs and then you need to find out uh your data set because you you need to find out what kind of load workload you are targeting for and find the uh find the relevant data set and then you want to evaluate those data sets to make sure uh it’s actually working for your need and then you need to find the environments and RL frameworks and eventually you need to connect everything. That’s not a easy job. And and and the industry Now, I think it’s

1:19:04 been moving that I think different layers are trying to getting closer to each other and so so the end users can have a like good experience. And also there are like more and more products that provide end-to-end experiences coming up. And also I think another important thing is uh previously people actually don’t know how to do that and don’t know or even don’t know what they need for that. But now I think there are more and more uh educations like the Hugging Face team has put out. So you like people can know what they need and

1:19:39 try to do that. Yeah. >> Yeah, thanks man. I mean there’s one thing also I’d add to that is like open source has really become like a default especially over the last year and so if you look at most product releases and everything that we’re involved in like everybody kind of comes together and that the redundancy of what of what whole org is doing I think is going down because people say right we will just integrate with you like okay come here I was like how do we get that to work and and frameworks are very very open now in a way that they they weren’t before. So you can plug different parts of the

1:20:10 stack together I think better than before. Um okay I’ve got another question but do any of you have a question that you want to ask each other or anything you want to bounce on from that? So the next question I want to ask is kind of inspired by Nathan Lambert. I ask this question a lot actually and he’s got this in this I think it was just a podcast he basically said like post training is like wrangling an org chart rather than wrangling a data set right? So it is really about getting the right set of people together into like the same room such that they one person

1:20:41 that actually understands the problem like the use case another person that can actually train a model and then maybe another person that can deal with the infra. Like could you just like kind of speak to like your own experience around that like how do you put one of those teams together maybe at like different scales like I imagine at the frontier it’s a really serious operation but like what does that look like in a startup like how can you speak to that? Maybe Victor you’re ready to go. >> Yeah yeah I mean absolutely I mean Yeah I mean definitely it’s like building a product right? You

1:21:11 in many ways people still still have to specialize in different ways and there are different reason why you you need a group of people right? I mean one of one of them might be it does a question like why cannot one person do everything? I mean I guess that’s also kind of your question to some extent. >> like why is it not a team of four MLEs like like what does the team look like? >> right so so what do you need not take for an MLE right? Again I think back to what I was just saying earlier the problem is that you’re building for something like you’re not And And not everybody has going to have the same skill set. Some

1:21:43 people are going to have to understand really deeply what you’re building for. And that is usually uh the kind of people that are going to have, of course, to be very sensitive to how you work with ML and understand very well like what MLE does, but their whole focus is going to be to understand I feel feel building for a client for example like really understanding what the client needs and frankly doing a lot of hand-holding. Just like mostly just doing hand-holding, understanding, and teaching, understanding what people have to The MLE is not going to do that, um possibly because it’s just not their job. They’re not trying to do so, but

1:22:16 you really cannot have If you don’t have this moment of transition where you don’t just learn You have to really know how to learn from a client. Let’s say we’re talking with this client. You have to really know how to learn from a client and how to explain and to make this transition, this is really a specific skill for example that that you have to put together, right? Um then you have probably like evaluation and training. I mean, those are really just competing interests like often. Like that’s why you do have the QC team that does something and a training team. I mean, otherwise you do have competition between a training team just

1:22:48 want to train just wanted to think out, you know, even if they’re very well motivated, it’s good to have a separation of concerns having someone else there, you know. Um so all these things you can do end up having like three, four, five roles of people that are just always Something with data. Someone’s going to have to worry about this and this is a whole different problem. And I think I what I’ve seen is that yes, you do need to have these different roles uh pretty much at any scale. And it’s really not unlike any sort of stack of any product building where, you know, you you do have um you cannot have a single person doing everything. I mean >> Okay.

1:23:19 >> That’s really cool. Go ahead, Govind. >> Yeah. Um sorry. >> Oh, you’re okay. >> Yeah, one of the things I’m starting to see is people organize in pods with FDs. You hear this in the news, but um having gone through with a few customers, I I think that’s the way work is going to evolve where you have a domain expert like I’ll I’ll tell you the business outcome that we got from from doing this hill climbing at this customer. They have this Let’s say it’s for it’s like a sales process it’s like sales enablement and we were able to essentially reduce it from like a few

1:23:50 weeks to basically a few minutes. And there was a lot of messy things that we had to work together to orchestrate and kind of improve upon. So, concretely like we had like couple of domain experts who actually knew where all the you know, access to tools and all of that was going to buried and how they do the work currently. And you don’t want to just automate the current process. That’s the wrong approach. It’s skeuomorphic. Like you

1:24:22 want to actually rethink what it is what is the artifacts that they are generating as part of that procedure. What are the you know, constraints and controls etc. that they need to care about. Like there’s a lot of design thinking that you need to first do to understand the problem space and then all the tasks and then and then you can like look at the evals that the person cares about and then like we basically created this using some heuristics some kind of um rewards. Like for example like some rewards where we want to reduce the

1:24:56 thinking budget for this tool not not important for the end output. So, there’s a lot of that weird hand stitching going on but that’s actually good because as a human like I mean I was a coder I’m not I’m not hand stitching code anymore but at the same time I have to upskill to complement this new world that we’re entering. I think that’s what it looks like it’s it’s not um well defined. Actually, it’s good that it’s not well defined and verified. And it’s a lot of coordination with yes

1:25:31 MLEs but also like you know, traditional IT security and like obviously domain experts things like that. I think like that’s how I’m starting to see some of these companies evolve into these pods where they’ll embed FDEs. And when I say FDEs, it doesn’t have to be external, like internal programmers that are not that are now been freed up to work on this approach. Yeah, that’s kind of how I’m seeing it. >> That’s really cool. Tao? >> Oh, yeah. Yeah, my view is a little bit different because first of all, I’m

1:26:01 Redstar is the AI infra company. So, basically all engineers here like are originally from like AI infra backgrounds. And but then when you start to building like infra seriously, you’ll find you need expertise in training, in data, and all those all those sorts of stuff. But I don’t we we don’t like hire a specific engineer with those backgrounds, but all the infra engineers start to become an expert in some area. So, I

1:26:33 think now because all the models are getting more and more capable, like many things are actually can learn can learn by people. So, a lot of those things you may you may start as an infra engineer, but then you may your scope become larger, and sometimes you even need to do some go-to-market stuff. But I think many of those things are actually learnable. So, I I think a team I think it’s fine with a team with skills same same skill set as a beginning as

1:27:06 long as like all the people love learning and can learn things and adopt new paradigms quickly. But probably I might be biased by my team because probably there are some areas you cannot you cannot learn or cannot be experts in a short time. In that case, it’s probably you still need some experts. >> Chenli, do you want to answer that one? >> Yeah, so I think there are two things that I see from a player perspective. So, I think one is uh So, one I think is a bottlenecks by

1:27:40 human attention. The other one is bottleneck by communication. So, what I mean by that? So, so for EOs, we worked with over 70 physicists on frontier physics. And we also work with external eval partners like they basically help us audit our benchmarks. Want to make sure when we release the benchmark, it’s like flawless. So, it usually goes like this. So, we have our internal quality checks. It like spits something and it takes a long time for physicists to actually read them, make sure there’s a lot of ASL. >> Okay. >> Sometimes there is a lot of ASL. And um

1:28:12 And then this is just internally. Externally, we work with like several eval partners. We give the data to them. They run their AI audits and we receive a spreadsheet like this is it. Each task fails. Actually, most are like false positives, but it took a lot of the time for us to like audit these. So, all the data collected in the process like the reviewers’ opinions, like the physics opinions, the domain experts’ expert opinion like matters a lot. But I don’t think in practice or in production there’s a way to collect all the subject matter expert experts’ opinion into the next generation. I don’t think

1:28:45 there’s a continual learning happening in improving uh automated like agents as a judge for for like the data side of the post training. And um I I I have some ideas on how to do that like sort of like how how do distill a set of reward rubrics to make sure you always arrive in with the same conclusion as the experts. But um yeah, that doesn’t seem really scalable. So, that brought me to the next uh idea about how do you uh basically increase the throughput by better communication.

1:29:17 So, like the agents when they read when they execute the traces execute the task they have traces, the reviewer when they review stuff they execute trace they generate traces. What if there’s a way for the reviewer to instead of looking at JSON files or like some viewers on dashboard? What if there’s a better way of communicating between the raw data and the reviewer cuz the reviewer they will know a lot of things about the task they created or the shape of the things they want. Um haven’t figured out that it’s uh like it like immediate medium is, but uh one

1:29:48 thing I see myself doing a lot of in production is so I use like 150 billion tokens for per past week. So one thing I really didn’t like to but I found a way to solve this sometimes the hey agents spin up 10 more sub agents and sometimes the agents will keep updating me in the main thread and not asking or answering questions that I have. So right now I just do this hey spin a sub agent that’s like a master agent for the rest of the agents and you just keep answering all the questions I have. So like this is a improvement on communication and this I think improved a lot of my productivity cuz like my

1:30:19 main thread just does not distracted. So I think something like this may happen in how the subject SMEs communicate with their agents communicate with you with you and you communicate with your your agents. >> Is anyone replies to that? I mean go ahead. >> Um I mean I think the we need to address like just the the vast diversity also of just like situation in which people can find themselves in. I mean when you are talking about what type of teams you need I’ve been at Sierra at Sierra we build

1:30:50 agents like just the whole point of the company. So we build a ton of them and we just that’s all we do for a ton of people putting them in production. You know so if you build a company like this of course building agents is sort of your brain but it’s just not one but just like for eight so you build a machine the machine to build agents that that’s like a totally different setup. Um at Mercury we build data different problems. You have companies where you’re going to have one agent in this case you know so where do you externalize when do you bring the capacity in who do you work with? So for

1:31:22 me whenever people talk about who do you bring in to do a task I mean the reality in any economic endeavor is always the question of when do you externalize when do you internalize it? I mean you have teams where you’re going to have like great people building you an amazing infra for data or for training and you know then you can really have maybe one person because everything is super well set up and yeah they can train model very easy places where you don’t have that in which case you have to either externalize or put a bunch of people together. So it’s It’s often an economic question honestly.

1:31:53 >> Yeah. So I mean, so I want to open up to questions in a sec, but something that it reminds me of is like my days of of teaching machine learning and like being very obsessed with the this matter cycle. I don’t know if you remember it. It’s like model, annotate, train, test, evaluate, revise. It’s like and when you’re dealing with like classifiers and regression models, that’s obviously like a cornerstone of how you build a data set, right? It doesn’t feel like we have a methodology, at least not a a public one that’s in literature, to like do this in the context of agents. Like there’s people who kind of theorizing around it, but I was recently writing like a

1:32:24 textbook for for post-training and and I I really couldn’t come up with a with a methodology that I would say like, “Oh, this is the the matter cycle of 2026, you know?” >> Yeah, I mean, that’s I think is very, very true. I mean, I just think we don’t really understand like I think the environment itself is an object that keeps evolving in terms of what it actually represents, whereas like I think before that we I mean, for me, we were in we were in ML maybe, I don’t know, not that old, but like 10 years ago. Um

1:32:54 we were talking about NLP, if people remember. >> Yeah. >> Um yeah, we were talking about like we had really fine we had come up with design patterns that worked, you know? You were doing things because we knew how to What did you do entity recognition? Nobody cares about it. It’s boring, but it works. That’s how we do it. That’s So, you there were also design patterns being put in place. You know, when when you were doing for me, entity recognition was really interesting because from the linguistic standpoint, all this it doesn’t really make sense. Nobody cares. Why do we do it? Well, it’s capitalized in English.

1:33:26 Uh it’s useful. It’s kind of easy semantically. We can recognize them. We know how to build stuff. So, we build algorithm, you know, hey, you know, you’re going to sequence recognition. So, you build your community around what we know we can do. What are the patterns? What are the objects? All etc. etc. Um and suddenly now we’re doing it we’re in a completely different world. We don’t know what’s easy, what’s hard. We haven’t settled on a steady state. So, fundamentally, I think like we have not we don’t we have not invented like yet those patterns. I I suspect you come back at this discussion maybe in a few years. People will be like, “Oh yeah,

1:33:58 you know, there are these type categories of problems that are we know pretty well. Those are pretty difficult, etc.” and then we will start having like develop patterns and I think open source community frankly and teaching Um that’s what I liked about Torbjörn and others just putting stuff out there and people learning and defining new methods and I think people are way ahead in their usage than most people like in the like who think about this problem really are. Like I think people are defining those usages right now. They are thinking about that. They are coming up with creative solutions to do this except they are still diffused everywhere and I think I personally

1:34:32 think this is where open source community can really help. People coordinating around these different problems cuz you know, this is always the story, right? You see people define usage, we centralize it and you know, that’s how we we move along. >> Yeah. >> Another sort of pattern that I’m seeing is like products that we’re building used to be for human developers. Obviously a lot more focus has been on being AX or agent experience and agent first, you know, shipping skills, shipping CLIs that are really agent friendly. I’m seeing this

1:35:06 in enterprises too where there’s focus obviously on MCP and in the physical world there’s a new standard that’s called MH something also from Entropic, but it’s to essentially make everything consumable by you know, an agent. That is been like this huge shift, right? So and and yes, that will automate a lot of the tasks that make up a job, but I think there are these new tasks that are emerging that we just have to like you know, adapt quickly and complement. >> Yeah.

1:35:39 >> Go ahead, Tao. >> I’m curious that like what do you think like what’s what do you think that what’s the difference between the product that I built for human and the product built for agents? >> Yeah, um good question. So obviously I I think for human we have to strive really hard to ensure that the human like understands it and you know there’s a whole you know developer audience teams and companies and uh you know docs and honestly our docs team is going through this crisis moment

1:36:10 right now because people just click copy markdown and they’re and they’re done. So I mean that’s a good question like obviously for humans. So I actually think there’s going to be a lot more uh videos. I’m trying to push GitHub to do this too where you know in every PR nobody’s reading the description. We’ll have a new experience next week. I don’t want to give it away but I do think like proof of work of a coding agent should probably be like a video explaining that it actually achieved the acceptance criteria and the outcome. Like because

1:36:42 because otherwise like we’re all operating at a much like a manager of manager of agents and sub agents, right? So imagine like 4 years ago if your VP of software engineering would care about a PR review or your code or whatever, right? They they don’t care. We’re all in the same like space right now. >> So >> Yeah. >> We are we are we are already doing that. >> Yeah. >> Yeah, that’s good. >> Big time. >> Yeah. >> No, I mean for me one thing I’m really curious is when you build a product and you know we work with agents we PR it’s always this

1:37:15 question of how much we talk about agent but it’s also like agentivity like how much you feel like you’re the thing you’re doing like this really belongs to you. Like you really feels like you’re control you controlling it, right? And I think um right now in coding at least in PRs and all thing, right? I mean for me it’s really funny cuz always like can you tell your cloud to review my Devin’s PR because my you know it’s just that’s how it works. Um so I think there’s going to be a switch between things like agentivity is linked to ownership. Like if you own something you should you should only you should be control of it because you’re legally responsible for

1:37:47 it. You get fired if you make a problem. Um I think some of the things we’re going to give up control on them. Like they’re going to be just not something we control anymore. We’re going to be like you know I don’t care that someone is reviewing this PR as a human. It’s just going to be reviewed and then we’re going to change this entire stack so that it’s basically fitting the elements that we’re working with that, which is like agents. And then we’re going to redefine what what what ownership like really means. Like what you’re supposed to own. And then products are supposed to are going to be supposed to give you the right levers so that you feel like that you really own and control you don’t feel alienated from their work. And I think this is

1:38:18 going to be like, you know, it’s going to take a bit, but I think this >> accountable, right? >> Yeah, so but we’re going to redefine what we’re going to be accountable on. You know, I mean, I think that’s that’s interesting. >> That’s really interesting. Okay, I want to open up with some questions. >> Let me preface by saying I I love post-training, but I’m going to make a case for not post-training just for the sake of discussion. Um So, um look, one, uh any environment that exists in the open the hill climbing happens for the Frontier Labs are going to hill climb. One. The cost of whatever model is being

1:38:50 released is keeping uh or intelligence per, you know, whatever uh cost it keeps going down. Second, the flood of data providers providing environments for what you can think about is just going to help like whatever you’re thinking. Four is post-training like is not static because what you think like customer behavior, let’s say you train a customer support agent. 2 months later, it’s going to change because agents are the ones interacting uh things change. So, in this world it just becomes so hard for somebody to be like, well, what is the value of me

1:39:25 post-training versus like, well, I’m going to use Haiku X or XYZ like next version and so on. So, would love to get thoughts on that. >> I’ll just go real quick. You’ll be shocked at how many customers are still using GPT-40 out there. We’ve been trying to get them telling them it’s getting deprecated. So, you may be right. However, most customers actually just care about their use case. If it’s working, they’re actually not going to touch it. And they’re actually asking us to please stop like uh deprecating models. So, that’s actually one of the reasons

1:39:58 they want to post-train and have a multi-water serving that is like theirs and not worry about it. Yeah, I hope I answered it. >> Yeah, I think another like important reason is cost. Yes, some like frontier models can handle your task, but if it’s a very frequent use case, for example, you are it’s a customer service agent, you probably don’t want to use cloud of office 5 55 forever. In this situation, you you

1:40:29 may find a very small model, but with post-training on your specific use case can provide similar experience or better experience than a frontier model, and this can reduce your cost dramatically. >> One thing I would like also to say is that I personally think like post-training or not is I mean, first the question is what are you trying to achieve? I mean, are you saying like a model is just going to be always good? Or I mean, what matters is that what can your model adapt? I mean, the environment is changing. Like it

1:41:01 changes. Like it does that adapting >> Yeah, your post-training model doesn’t necessarily change, and that’s hard to change versus >> Sure. >> the other models that are open, you know, frontier models, they keep evolving. >> Right. So, one thing >> Interface switching is so easy. >> Yes, except then what I would say that I think people vastly underestimate how complicated it is to sample the entirety of the real world and the real I mean, if you do B2B, you know that every business is just weirder than the next. Uh, so I would say yes, but then what you would need is to make sure those

1:41:33 models can be highly contextualized for your particular business at this particular time and this particular moment, etc., etc. So, if we are able to design systems such that all this information can be ingested, used by the model like at inference time, so that it’s always understanding your very, very local like parameters, fine. Um, my feeling is that that’s going to be expensive. That’s going to be a lot of token. And so, at this point you’re going to have the typical trade-off, you know, like do you do you shove that somewhere in the prompt and just be done with it and do some context management, or do you post-train? When you need latency, you got to post-train.

1:42:07 When you do cost, you got to post rate. Well, so it I think it becomes like that’s going to be the trade-off. Nobody’s going to care whether you post rate or not. They’re going to care whether it works or not. And the question is like do you have the money? Do you have the That’s it. >> Cool. Next question. >> We’ve got another question here. >> Oh, yeah. I have I just have one thing to add. So, I feel the ultimate goal of post training or like ultimate goal in this cycle should be we should be able to post rate an open model that is on par with open siphon five capabilities. If we can start this with one spikiness

1:42:39 of that capability, maybe RSI, maybe design, maybe cyber. And if we can prove on this spikiness, we can go for all spikiness cuz it just So, yeah, but but I I I I I They are like generations ahead, but um I think I think it’s possible. >> Yeah, my question kind of echoes, you know, the previous question a little bit. Uh so, I’m actually coming from this other conference and down in San Jose at Gen T&A conference, right? And the observation that I would like to make is that post training wasn’t really

1:43:10 mentioned once, you know, during the day. Now, again, it’s a sampling of one person, but and I think I know why because, you know, the customers basically have this pipeline, you know, the enterprise customers. I’m not talking about, you know, Silicon Valley, you know, even like just San Francisco companies. They have this pipeline that the vendors and now there’s a huge constituency of vendors. Like the the conference is, you know, like apparently, you know, 5, 6,000 people, right? All of these vendors are pushing one particular way of how enterprises are

1:43:40 supposed to ingest AI and it seems to be resonating, right? So, again, like the customers basically like if you start talking to them about something else, they’re like, “Well, yeah, maybe that’s like my problem in 3 years, right?” So, I’m just wondering, I mean, do you guys actually have success and especially for Govind, you know, since you seems to have the most experience working with enterprise customers, do you actually find success in finding something that really resonates and cuts through the noise that all of the other vendors in the ecosystem create and basically like, “Don’t worry about post-training. Like that’s that’s not

1:44:12 your worry. That’s too much complexity. >> So, the funny thing is the most So, I also work with ISVs who create vertically specialized solutions, as you mentioned vendors. They’re the ones that are interested in post-training. Because for them, the biggest cogs is frontier model. So, um I think I think it’ll start with them. But then what what I’m seeing with enterprises is there there’s actually especially now um I I think it’s going to get more intense is uh de-risking anything with the model.

1:44:44 Again, my CEO keeps tweeting about this every week, so it’s top of mind for our company. But like essentially operating in a world where you have zero trust to the model, and assuming that the model is um or really the agent uh because you know, if you have a ISV agent, right? Uh and assuming that it um it is like a it is like insider risk. Like if you’re a CISO, well, you’ll understand what that is, right? Like basically, treat it like it will, you know, attack. So, give it least privilege, zero trust, all that stuff. So, yeah, to your question, um

1:45:20 I think yes, inertia is real, but the vendors that are creating vertical solutions that actually solve a business problem, the way it’s going to the way it’s working is like they will, you know, sell this package solution, and then they have every incentive to reduce their costs, uh because that’s the margin. So, they are the ones who are going to start post-training. >> Cool. Do we have any other questions out there? I reckon we can get two more. I know I saw your hand on the first get-go, so I’ll give you that. >> Thanks. >> Um I was just curious in terms of the

1:45:53 whole pipeline of post-training, uh where how does uh where are we where is like the ecosystem in terms of like continual learning and like maybe like creating synthetic data sets automatically and like continuously improving the system. Maybe even benchmark creating benchmarks automatically. Like how close are we to that? Cuz people were talking about this like four or five years ago. >> So, I think what the industry So, there is a blog post by Ophir Press on like human-level evals or human-level AI evals are are over. Did you see that?

1:46:30 So, basically means uh So, so like if you take SweetBench, SweetBench basically is a set of tasks that um maybe take an engineer a few hours to fix. And um similar for Terminal Bench 2. If you take Terminal Bench 4 or uh Program Bench. Now, Terminal Bench 4, it may be like it’s a task that takes you a week to solve. So, some some of them including, you know, uh re- re-implement some algorithm in a paper or something like that. Um So, something in Program Bench basically reverse it binary of

1:47:01 Do you know what Program Bench is? So, Program Bench is say I give you the documents of SQLite. I give you all the feature that you should have. I do not give you the source code. I give you the binary. So, you can test the binary. You cannot decompile it. Can you rebuild the binary? For example, sort of sort of speak like can you rebuild Linux? Can you rebuild Chrome? So, like push alarms to do that. So, basically this is the generation of evals. I think Program Bench is cited profusely by the Opus model card. And this benchmark has a trade of

1:47:33 superhuman level intelligence cuz it will take teams like months to rebuild this. And um I think we are at the edge of like the next generation of evals. So, like So, if you look at Terminal Bench, their audited report is it’s like 90 89% solved by Sonnet 5.5. Basically, if if there is no ambiguity in the task, Sonnet 5.5 just almost all shot one shot every task. And I think that is because there isn’t enough human hours in this or this benchmark is not

1:48:04 superhuman enough. And um to like fundamentally create a superhuman level task, you either have to This either either can be super verifiable like a Navier-Stokes. The Navier-Stokes can be a benchmark task, right? Another thing is like how can you reliably compress like a lot of human data into this one goal, one task. So, this is something I’ve been thinking a lot about. Like, there’s a lot of data vendors who’s like package your company data, sell resell to Frontier Labs. So, basically your company’s past 10 years of data. So,

1:48:36 like my guess is in the next 3 months the the the data would or like the data one company generates in 10 days will be more data than a company generates in the past 10 years. So, like we sort of have to think in that way for eval and training data. So, like basically in short, more ambitious. >> Yeah, I I see like continuous learning as a quite overloaded terminology. And I think the most ideal way of continuous learning is you have models that con- continuously updating its weights based on the new

1:49:06 user interactions. And I don’t think like currently our people are doing that. I think it’s less technical reason, but more it’s a business decision or legal compliance stuff. And I think today what’s like the common way is we still have a static model for to serve all the users. But in the process of the like in the process of the developing that static model, we may use the latest capability of this model to to make the model better and to use the

1:49:40 latest model to help the or the development process faster. That’s happening. >> Um I mean, I’m not sure maybe for the question. I mean, for me you were asking about continuous learning and maybe models building their own training data or I mean for me it’s always a question like in what situation? I mean, we hear a lot about >> What if you had a big model and a small model that you wanted >> No, so the question becomes exactly. So, when you use when you’re at Sierra, you build customer support. If I ask people how kind of like superhuman intelligence is required to

1:50:13 like solve a customer support call? Probably not a lot. I mean, you know, like it’s not a super intelligence task and most of the task I do are not super intelligence task. I mean, I think there is two world. There’s the people who care about super intelligence should care about this. It’s very interesting. It’s solving any problems. And then there is the rest of the world that thinks about other stuff, right? Like and this is a very different problem. If you ask, can we do benchmark to like really help fine-tune a model locally for problem that’s not to go It’s really about like local sampling.

1:50:45 How do you get there? What do you train on? That’s one very complicated problem, but very different from like can you do like AGI self-improvement? I mean, the way I like to think about it is at some point people develop the motor, the engine. At some point we’re like, “Hey, we can build cars out of that.” And we let’s make them a bit better. Cool. But then people start driving them and then people are like, “We’re going to build an engine to go to the moon and to Mars.” And they’re like, “Good. Just do that. That’s super cool. But meanwhile, like let’s make the cars better.” You know what I mean? Like we don’t need like an engine to go to the moon to make a car better, but we do need many, many,

1:51:18 many, many other things including infrastructure and whatnot to make that happen, right? So fundamentally, like I think there’s just two different worlds and I think we really always need to think like what world are we talking about? Are we talking about the world of like a regular enterprise that want to make better like data to make that model better so that they can do better Excel sheets and stuff like this versus like AGI. I mean, I think this is just vastly different. >> Yeah. Um regarding continual learning specifically, like the one thing that I’ve seen that uh work and I think it’s going to become more mainstream is like having a graph representation of

1:51:52 a user or a certain task and then having like this dreaming functionality or like a compaction function functionality. Um obviously, that’s all harness level optimization, but I think that’s a good start. The one that really showcased a good vision is actually Apple. I don’t know if you all remember 2 years ago when they talked about Apple intelligence AI. And then um they added these adapters. For some reason, they could never pull it off, I think because it’s very complicated. Uh but

1:52:24 that’s that’s probably the vision for multi Laura serving. It’s like, you know, you have obviously cursor, you know, did that. We did that at with GitHub co-pilot, too, but you know, whatever. Sorry. Uh but I think um everybody can start with like a basic graph database, you know, having a dreaming functionality that runs every night that consolidates. >> But the other things are still very hard like users. We we don’t like simulating If you want to do real like data for real people, like if you ask tell me like I work a lot on user simulation. People

1:52:57 I mean, models are are not very good. Like they’re just not very good. Like most of the data right now is just not like if you try to condition a model to really act like a particular person and really generate something, this is not very good. Um for many reasons. I mean, they’re not trained to do that. They you know, they I mean, we are individuals that are just mostly just not knowing things as opposed to models that are mostly just knowing everything. So, there’s quite a gap. But you know, like not very good. Like most studies would show like, well, they don’t do better even if you give a whole history. They don’t do better than like if you give demographics or things like So, that shows into training data you’re going to create. Like it’s going

1:53:29 to feel sloppy because it’s not going to feel like you’re actually working with this particular person or this type of person. And so, okay, yeah. Like I think a lot of people think coding, think math. Those are problems that are like chess. You know, like they’re not people. And there are people in coding actually. Plenty, plenty of people in coding, but some problems are very people heavy like in terms of what you have to solve. And we don’t really know how to simulate like people very well. We don’t really know how to near that. So, for example, that’s a pretty big problems, but I don’t think I think we have a problem in thinking of this as intelligence. I think we think of intelligence in terms of

1:54:01 very complex problem that historical kind of solving, but we don’t I work on conversational systems. And I mean, just taking turns at the right pace with someone like that’s really hard. We do it at the sub-millisecond capabilities. I mean, models don’t really do that very well. Okay. I mean, you know, it’s classic idea that is like it’s easier for models to to learn how to do do a move in chess than to actually move the piece of chess. Like this is just harder, right? So, I think we still need to focus on the real world, what people really care about, look at this data, and I think try to

1:54:32 really make better here. And I think we’ll see there are still a lot of gaps into making that happen. >> All right. Can we take this last question then? That would be >> Yeah, good. >> Thanks. >> Um I have a lower-level more pragmatic question about each of you as individuals and your observations seeing as most of the execution of training and writing code, etc., is being handed off to agents. When you’re on the absolute frontier and you’re faced with a particular problem, I think most of what you’re doing are experiments. So, you’re

1:55:02 observing information and you need to make decisions. Um what are you reaching for? Are you mostly hunched over in an IDE? I imagine that’s not most of your work. Are you reaching for pen and paper? For example, you said that you were observing something where maybe you would want to speak with a sub-agent to have better inter-agent communication. You have an idea for a new experiment. What are you reaching for in that moment? >> Uh uh so, that’s a good question. Uh right now it’s for like RSI or MTS or

1:55:34 training more model stuff related. Agents aren’t really good. Even Opus 5.5 is not that good. Like one example is that, “Hey, I tell it to Hey, can you like stack up the Slurm jobs?” So, Slurm is a GPU job scheduler, so that GPU never goes like idle. It It almost never does that. And um and also like when should I get expect training signals? So, it it told me like 12 steps, but uh hey, I just checked these slides. I asked the people at uh at at some like large data company where at some frontier lab is that, “Hey, at least 100 steps.” So, like asking other people, and I’m sure you

1:56:07 all do this a lot. You take a screenshot of your terminal or like your T3 code or your cloud and take the screenshot asking Slack, “Hey, like is this Does it Does it make sense?” So, I think that’s one thing I observed. And then there’s actually another case where you mentioned video, so I basically in In physics, we observed that in one task, agents just outperform human at all times. And uh we want to see if there are more patterns like this in physics. So, we just basically created like So, like in physics it’s not like mathematics. It physics does not have these a lot of open well-defined

1:56:40 problems. So, we just have to find a list of problems and we try to push like agents each give them like 40 hours to uh try to spin up a recursively sub agents to solve it. So, we basically have like 13 things 13 ish task that we think it’s kind of like beats human soda. So, the way we sent to uh human reviewers is that there is going to be a paper we generate. So, in frontier physics, we also evaluate So, the task itself also ask people to agent to generate a paper cuz that’s what’s going to human going to read. So, in addition to this paper, we generate a video in the uh 3Blue1Brown style. So, this

1:57:13 actually give the reviewers a very easy time to review. I’m not sure if the like video between people is the right way to communicate because some of the videos I generate to for for myself to watch, they’re like a bit redundant and the agents are like So, so, so, so like if you have to think about this, when I’m when I’m making the slides like it is So, when I have to think about the communication communication problem, I basically at least like there is a 50 prompts I uh I send to the agent to edit the slides before it’s this shape. So, I

1:57:43 feel like this is still a hard problem to solve. And this is also why I think I could go back to the other question. I think should be in sort of some some sort of form of continuous continuous learning cuz there’s no way that my 450 billion tokens per week whole company just goes nowhere. And I think they should go into some model weights. If it’s not the closed models, it should be some open models. And uh no, I do not think you can just update a database and model just gets smarter. So, yeah, not sure if that answers. >> Cool. Okay, I think we’re going to wrap it up