AI Engineer Paris 2026 Main Stage: Google DeepMind, ElevenLabs, Hugging Face & Stripe | Day 2

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: AI Engineer Paris 2026 Main Stage: Google DeepMind, ElevenLabs, Hugging Face & Stripe | Day 2 · Channel: AI Engineer · 8:42:48 · Watch on YouTube

Topics: AI agents, Enterprise AI applications, Model inference and optimization, Multimodal models, Robotics and autonomous systems

Summary

Day 2 of AI Engineer Paris 2026 features talks on adapting generative models for robotics, scaling voice identities, running Hugging Face Transformers models in vLLM, and the changing economics of AI companies. Speakers explore real-time speaker diarization, infrastructure for training and inference, low-latency serving, edge AI, and agentic methods for analyzing large codebases. Later sessions cover improving PR review workflows and efficiency techniques in Gemma models, followed by a discussion of conversational AI and a report on workshops that introduced children to coding with AI. The conference closes with thanks to attendees and sponsors.

Key points

Black Forest Labs describes customization as a way to adapt Flux through prompt upsampling, moderation, fine-tuning, and new capabilities such as robot action prediction.
ElevenLabs built a pipeline that maps voice characteristics to visual orbs, using automated generation, review tools, and shared services to scale the system across products.
The vLLM Transformers modeling backend combines model definitions and weights from Transformers with vLLM’s runtime and performance-oriented modules.
Stripe’s data points to faster AI company growth, international sales from launch, evolving usage-based and hybrid pricing, agent-oriented go-to-market, and the need to address fraud early.
Real-time speaker diarization differs from batch processing and requires balancing latency, accuracy, and serving cost; the talk describes a system with roughly 400 milliseconds of latency.
Mistral and NVIDIA discuss agentic workloads, the roles of CPUs and GPUs, rack-scale systems, infrastructure flexibility, and growing inference demand.
Modal’s inference optimization approach prioritizes understanding workloads, speculative decoding, quantization, host-side bottlenecks, and routing before kernel-level tuning.
ARM highlights how smaller open-weight models and accessible hardware enable edge applications, including interactive devices and hybrid on-device/cloud inference.
Cognition presents an agentic map-reduce approach to codebase-wide tasks, splitting analysis into bounded areas for parallel agents and consolidating their findings.
Matt Pocock advocates layered checks, automated code review, human review, and retrospective skills to improve PR quality without overwhelming reviewers.
Gemma’s efficiency techniques include per-layer embeddings, tailored quantization, speculative decoding, encoder-free multimodal processing, and diffusion-based generation.
Gradium discusses speech-to-speech models, real-time serving, distributed infrastructure, and small speech models that can run on devices.
The kids workshop used Mistral Vibe and a guided repository setup to help children build games with AI, while emphasizing accessibility and inclusion.

Chapters

Start	Chapter	Summary
16:15	Welcome and Flux customization for robotics	The MC opens day two, then Black Forest Labs explains how Flux can be customized for media generation, moderation, and robot action prediction.
42:20	ElevenLabs’ voice-driven visual orbs	Dorian Lods describes building a scalable system that turns voice characteristics into branded, interactive visuals.
1:08:03	Loading Transformers models in vLLM	Hugging Face engineer Harry Miller explains how vLLM integrates Transformers model definitions while applying its own runtime and performance features.
1:30:59	Break	The MC closes the morning session and announces a break before the afternoon talks.
2:18:56	Stripe on the evolving AI business playbook	Ariel Loubai shares Stripe observations on AI company growth, global sales, pricing, go-to-market changes, and fraud risks.
3:11:58	Real-time speaker diarization	The Piano AI co-founder explains speaker diarization and the engineering trade-offs involved in making it work in real time.
4:50:11	Mistral and NVIDIA on AI infrastructure	A fireside chat examines agentic workloads, compute requirements, rack-scale systems, infrastructure reuse, inference economics, and European AI factories.
5:17:45	Optimizing low-latency inference	Modal’s Charles Fry presents an optimization playbook covering workload analysis, speculative decoding, quantization, CPU bottlenecks, and routing.
5:43:42	Edge inference with ARM	Dominic Payak discusses on-device AI, efficient models, interactive hardware examples, and ways to route tasks between edge devices and larger systems.
6:06:48	Agentic map-reduce for large codebases	Cognition’s Yanis Sorakis explains how parallel, bounded agents and a reduction step can tackle codebase-wide tasks such as security analysis.
6:25:11	Break	The MC thanks the audience and announces a break before the final stretch of talks.
7:14:50	Fixing the PR review bottleneck	Matt Pocock proposes layered automated checks, agent-based code review, human review, and retrospectives to improve PR quality and review speed.
8:03:30	Gemma efficiency and open models	Martin outlines Gemma efficiency techniques spanning embeddings, quantization, speculative decoding, encoder-free processing, and diffusion.
8:06:37	Conversational AI and voice models	Gradium’s Olivia Tubul discusses speech-to-speech systems, real-time serving challenges, distributed infrastructure, and small on-device models.
8:30:54	Kids workshops and conference close	Cassandra Chin describes teaching children to build games with AI, followed by closing thanks and a brief Backblaze sponsor presentation.

Quotes

it’s super quickly to iterate — 28:48

as specific as needed and as general as possible — 41:50

The reviewer should commit. — 7:33:05

by real time we mean less than 300 milliseconds. — 8:22:31

Concepts and entities

Flux, Black Forest Labs, ElevenLabs, Hugging Face, vLLM, Stripe, Piano AI, Modal, Cognition, Gemma 4

Full transcript (original language)

16:15 Ladies and gentlemen, please join me in welcoming to the stage your MC for the AI engineer Paris 2026, developer relations engineer [music] at Replet, Rou Chevrey. Good morning. M check. Good morning everybody. How’s it going? Hey, this feels like day two energy, so we need to bring it up. Okay. How’s it

16:46 going everybody? >> Yeah. Nice, nice, nice. Well, welcome to AI Engineer Paris day two. All righty. So, who was with us yesterday? Show hands. Okay, most of you guys. That’s awesome. So yesterday was all about how to think about the evolution of technology to think about AI today and then we talked about the future of AI right the future of AI um software factories um and also uh the the future of frontier models but I’m particularly

17:17 excited about today today we’re going to talk about generative media about software factories again we’re going to talk about uh graph rags and we’re and we’re going to cover so much more so So with that, I want you to uh think about what you want to see. Please pay attention to everything that is going to be said here. Our speakers are going to be hanging out. So also take the time and go to the expo and talk to everybody to learn as much as you can. All right.

17:47 So you know the drill by now if you were here, right? This event wouldn’t be possible without Mrol. So I would like to give give it up for them. So please let’s give it up for M. Thanks for organizing and let’s give it up for our sponsors, our platinum sponsor, Nvidia. Yes, please. Yeah, keep it going. So, to for all our uh gold sponsors, silver sponsors, bronze sponsors, and also our supporting sponsors. Let’s give it up one more time. Yeah, thank you. Okay. All right. So, without further

18:18 ado, our next speaker is from Black Forest Labs, and he’s got some tendency. He loves going to the dentist. I don’t know why. Maybe we can ask him later. But he’s going to talk to us about um about how they went from image generation to robotics. So, please join me in welcoming to the stage Yakob Porchman. Thank you very much. Super nice to meet

18:50 you everyone. Just need my slides maybe in here. Awesome. Nice to meet you everyone. My name is Jakob. I lead the AI solutions team. Um that’s the four deployed engineering team and solutions engineering team at Black Forest Labs. And I would like to start in with a hopefully trivial question. So give me a brief show of hands of who has generated an image or a video before using AI. Just give me a brief show of hands. This should be trivial. Come on. This is

19:22 almost everyone hopefully. Um and um then hopefully if you generate an image or video before um you hopefully were in contact with Flux uh or Blackfire Labs before because Blackfire Labs at Blackfire Labs we are a research lab based in Germany and we build foundational image and video models for image generation. Um and you can see some of the samples um videos and images that were generated with our recent model flux 3 in the back uh back here. Now the second question that I have for

19:52 you is who of you would believe me um if I would tell you that you can use that same video model to also control robots and play video games. Okay, that’s a few people, not really everyone. Um well, it’s kind of a it’s kind of a rhetorical question obviously. um otherwise I wouldn’t talk about it because that’s exactly the final thing that I’m going to close with today is how we’re going to customize Flux um to not just generate media, generate images, generate videos and and edit um edit both um but we’re going to talk

20:25 about how to steer video games and steer robots with it. So you can see two different samples here that were also on the left generated with Flux. On the left you have Flux actually playing a video game. Um so this is not just a video of a generated but it’s actually flux taking the control and and generating the frames that are happening. And on the right you have a Lero that is completing pick and place tasks um with Flux the video our video model flux 3 as a

20:55 intelligence backbone and what is in between video generation and using flux as a backbone for a robot that’s exactly what we’re going to talk about today because the part that is in between that’s basically bridging that gap is customization of this flux model right there is a super powerful model that we pre-trained trained mid-trained on essentially understanding the visual world having having an understanding for um how objects are behaving in the visual world and we are taking this model and customizing it for certain use

21:27 cases and that’s exactly what we do in our AI solutions team as well and this is what I want to talk to you about um in a couple different steps to break this down a little bit before we go into robots that’s going to be the final stage of what we’re going to talk about um I want to talk a little bit about the anatomy of the inference pipeline because this is also going to give you an overview of sort of the customization vectors that we have when we walk work with flux and we talk to enterprise customers that want to use flux for very

21:58 specific use cases. Um this breaks down in approximately six steps. First of course there is a user input uh that can be optimized. Um this is usually a text prompt, an image prompt, a video prompt, fairly straightforward, right? Um second step is the prompt upsampling. Prompt upsampling essentially means that we have an LLM or a VLM that sits between the user prompt that is sent to us and the flux model. And we essentially try to understand the intent. We try to understand what the user actually wants to do and we try to improve the prompt

22:30 that the model generates something that’s closer to what the user is actually intending to do. Then there’s moderation. And you might be surprised in seeing moderation when it when it comes to when it comes to customization of a model. Well, actually moderation is one of the most important product surfaces. Um because moderation essentially decides what you’re going to see or what you’re not going to see. It’s not part of mostly not part of the model. Um but it is part of the user experience and we’re going to talk a little bit about how we customize it and also how we actually embed moderation into the model weights. Fourth, there’s

23:02 of course the model. There are the model weights which we can fine-tune. We can run laws on it. We can fine-tune for style, for characters, for products. And we can fine-tune also for behavior and using fine-tunes, we can also add whole new modalities. And that’s exactly what we did with flux flux 3 action, which I’m going to talk about in the end of my presentation. Then there’s a post-processing step. Of course, this is kind of arbitrary. There can be upscaling, there can be restoration, there can be further moderation, right? Um, endless customization opportunities here. And then finally there is um we

23:35 also work a lot about how we actually serve the model, how we deploy it, how we optimize it, how we distill it um etc etc. And what I want to do in the next 20-ish minutes is I want to go through essentially the four deployed engineers toolbox at least scratch the surface a little bit um of how we customize flux um in our day-to-day. So starting uh with the first major customization I want to focus on this prompt upsampling step. As I already mentioned prompt upsambling means that there is a VLM or

24:07 an LLM that sits between the user and the model. Um that looks in practice something like this. Um so on the left here you have a user prompt. So this might be super super straightforward. a cabin in the forest. And if you send that straight to the flux model, generate an image, you’ll see something like the image on the left. It’s a plain um cabin in the forest, right? The mission completed. However, it might not be optimized for user intent. It might not be optimized for user preferences. Um it highly depends, of course, what the user wants

24:37 here. In this case, the prompt is fairly simple. Um but it demonstrates the purpose quite well. So essentially we take the cabin in the forest and we essentially extend the prompt. In this case the default upsampler just optimizes for user preference. Extends the prompt the user the cabin in the forest to something like a hyperrealistic right angle stone chimney softarm golden light etc etc. And then you see the final result um um on the right. Whether you like the left or the right better, that’s up for up for discussion, of course. But the point

25:09 being here is that the prompt upsling step gives us an incredibly powerful tool to actually steer the user intent and the generation that we’re taking into any direction possible. So let’s take a look at a concrete um concrete case. So this specific this was a real customer case. The customer essentially came to us and told us well actually we want to illustrate certain conversations that a user having and users are having in our application with flux images. So they want to create illustrations for topics. These topics could be completely

25:41 random and they’re machine generated. So they’re not hunt created and they’re also not image prompts. So for example this could be familyfriendly travel destinations or draining water for my dishwasher. Right? Completely random topics not in the form of an image prompt machine generated. uh this could be a vast variety of topics. Now our task is to generate an illustration somehow illustrate this nicely. But the trick to this is that of course there are certain design requirements that we need to meet. So for example um in this specific case the subject is supposed to

26:12 be on the right. It’s supposed to be a very clean image. There’s supposed to be a lot of space on the left for a text overlay. And there are certain design rules that also apply. For example, no people involved, no landmarks, and a couple others. So what we essentially do or what this actually comes down to is that the model the image model is actually not the bottleneck anymore because the model technically can generate all of these all of these different different things, right? No problem. But really the design requirements end up being the bottleneck and where prompt assembling comes in is

26:44 matching and bridging the user prompt with the design requirements and matching these design requirements to the training distribution of flux. So finding the exact right spot that we need to send to the model um from the training distribution to generate the the exact image um that um slides are gone um here we go um to generate the exact image that actually follows the design requirements. So to give you two examples from the the

27:15 topics that you already saw, um first they’re familyfriendly travel destinations. Um if you send that as a raw prompt to flux, completely unseampled, you’re going to see complete gibberish, complete uh trash trash images essentially. For example, there’s this this postcard style. The v variety between different seats also is extremely high when you just send this completely unupled. If we use our default upsampler, which is just optimized for general user user preference, then you you’ll see something like in the middle, probably a

27:46 little bit closer to somewhat of an illustration, but not at all matching the design requirements. But if we actually customize the upsampling step, and we embed the design requirements directly in the prompt upsampling, you’ll see something on the right. So the subject is on the right. It’s a very clean image, very sort of glossy. There’s a lot of space on the left for text overlay. So meeting exactly the design requirements. Same goes for the second example. So draining water in a dishwasher without propped up sampling complete complete random image. Um

28:16 getting a little bit better some sort of instruction video but still not at all what we want in the second case. But if we actually customize this upsampler which is essentially the system prompt that looks at the intent and tries to match that into the training distribution knowing the design requirements. we’ll get a very nice, very clean image, subject on the right, a lot of space on the left, text overlay possible. Um, and this is essentially a pretty simple trick. Um, because the um the massive example and what we essentially learned from this um the

28:48 massive advantage is that it’s super quickly to iterate, right? You essentially just iterate over a system prompt. There’s no training needed. Um, and you have maximum use case flexibility. And this is sort of necessary as well to include the step because what we learn especially with when talking with enterprise customers is that users are just lazy. Users don’t necessarily want to know exactly how our training distribution looks like right how to perfectly um perfectly prompt flux. That’s in a lot of cases where we come in and we customize prompt upsampling to match exactly user intent

29:21 design requirements and training distribution to find exactly the right sweet spot here. Um yeah, front up sling seems super simple but it’s an incredibly powerful tool and because it is so simple to to iterate over and it doesn’t need any training. Moving on to the next stop is moderation and safety. And you might be again a little bit surprised to see this in the customization surface here, but moderation is actually one of the most important product surfaces that we have um especially when offering a model to

29:52 the through the API because essentially moderation decides on what the user sees and what not, what which prompts we allow and which not. Um so this is an incredibly important piece of the user experience that we offer. If you have tried some video models um state-of-the-art video models in the past, you might be very aware of error messages like this. You put in an image of yourself and you want to animate yourself, but you’re getting something like the images of videos might find likenesses of real people, etc. This is obviously not a model limitation. This is purely moderation service, right? That doesn’t allow you to generate

30:23 something. And when talking to larger customers that want to do very specific thing thinks this is a massive issue and we need to customize and solve for this. Um, so two ways that I would like to talk about moderation. First is the most simple way and it is essentially in the configs. Um we have a modular setup to essentially customize configs. There is a fairly straightforward safety parameter here um that allows you to set that via API. And when customizing uh

30:55 when customizing moderation for specific customers, we have a very fine grained confict setup that essentially allows us to set safety tolerance but also allow or block certain um certain categories specifically. So violence, sexual prompts, self harm, IP and likeness. And we also combine that with certain block lists and allow lists that we can set specifically for every customer. So think about, for example, a kids platform being much more strict, blocking violent, sexual, self harm, IP likeness, obviously, and then having

31:25 even a specific block list. Uh fashion retailer being a little bit more a little bit more open, maybe allowing a little bit more of sexual leaning prompts. Um but blocking specifically competitor brands and allowing the own portfolio essentially only trying to generate their own their own stuff, of course, right? And then there might be a media or a movie studio that uh is much more flexible, allows even some violence and specifically wants to allow certain licensed characters even if they’re AP

31:56 protected and wants to block every other character that is not that they don’t have a license for. So this is fairly straightforward. We just set the config super modular, super simple. However, this doesn’t work for every case um because we also deploy our models directly on customers infrastructure. So in these cases, we don’t ship the entire moderation stack. we somehow need to find a way to actually embed the moderation and the safety into the model rates. And this is where for example um a slider law comes in which is the next sort of tool in the toolbox of um our

32:28 FTEEs. A slider law tries to learn an abstract concept or tries to learn the specific abstract con concept that we’re trying to block and tries to then turn around the rates that it learned to block that specifically. So let’s take the example of a desexualization slider lower um which you have essentially simplified here on the slide. Essentially you try to tweak the model tweak the lower adaptation matrix you try to tweak that into the d the rates into the direction of the of the

32:59 pure concept of sexualization. How we do this is that you have a generic data set of images for example of people in this case one image of a woman in a black dress and you feed that to every um training loop tries once with a modest caption and once with a sexualized caption. So then you essentially train the model. This is the image and you see it tries one once modest and once sexualized and what what what is left in the rates is essentially the delta between [clears throat] both learnings. And the delta between both learnings is

33:29 both both captions essentially contain there’s a woman, black hair, black death, whatever. But the delta between those is essentially the concept of sexualization because the sexualized caption contains that but the modest caption doesn’t. It’s much more descriptive. So you end up with a law matrix with an adaptation matrix that essentially describes the concept of sexualization, right? Um and in inference we can now go ahead and apply that to our model weights with a negative alpha. So we turn the weights

34:01 exactly around to go in exactly the opposite direction which then turned out in this specific case for uh for minus 20 nudity detections and um 68% judged less less sexualized image quality. while at the same time the image quality generally is a complete tie because we literally only turned around the concept of sexualization. Um so this is a nice tweak to really bake in moderation into the weights and this is something that you can also ship directly to customer they can deploy it on their own infrastructure um to have a much safer

34:34 model without any deterministic uh moderation around it. So what we learned from the moderation piece is that first of all safety is an extreme extremely important product product surface. Um it’s essentially the first thing that users interact with in some sense. It decides what users see what users don’t see and it can be a USB. It can even be a business model in certain certain cases. Modularity wins. Baking in moderation into model rates is cool. It’s fun. It’s a nice concept to build some sliders. However, it’s

35:06 expensive to train. it’s static, you need to update it somehow. So in many cases or in most cases, we just go with simple confict config customizations because it’s just modular. It’s easy. It’s scalable, right? Um it’s it’s much cheaper to actually set up. Um and yeah, as as I already mentioned, motivation and weights is possible. However, it is expensive and static. All right, let’s move on. So, we already looked at the fine tune that changes behavior in the moderation direction, but I promised you

35:37 some robots and I promised you some um flux controlling some things. So, finally, we’re going to look at customization of model weights um and how to fine-tune flux for specifically action prediction. Um just to give you an overview, generally if you fine-tune a model, usually we come with a certain goal in mind. If we’re talking about video models or or image models, we either fine-tune for a certain style, we fine-tune for a certain object that we want to present or a set of objects. We fine tune for a certain character that needs to be um extremely accurately um

36:08 displayed or we fine tune for behavior. We also have some models that we find in for example outpainting, specific local editing, desexualization, right? All of these are behavioral finetunes. And then if you go even further, you can even think about fine-tuning for certain to to add certain modalities if we even go a little bit into mid-training and post- training. Again, that’s exactly what we did with Flux action. Flux action is a new um version of Flux uh Flux Backbone that we um continued mid-training and postrained and fine-tuned for

36:40 specifically action prediction. This was actually released into open weights yesterday night. So check it out. It’s brand new. It’s in hugging phase. You can try it out. pull the rates and control your own layer robot with it. But just to give you a brief rundown on how it works. So we have an image model that essentially predicts frames um predicts video frames based on um based on a text input mostly, right? Um so how flux action works is that it has a text conditioning and it has a video conditioning. In this specific case, we work with a LE robot that completes pick

37:12 and place task. The robot in this case has two camera sets. one is at the uh grabber hand, one is above um the top camera essentially. At the same time, we also fine-tuned flux action for the um for the embodiment of of the SL101, the Lobot robot arm specifically. So we actually teach flux that this specific robot has six axes and this the the the movement of these axes essentially represented by a vector that in every

37:42 frame defines the direction that every axis needs to move into. So essentially is simplified a sixdimensional vector that we’re trying to predict here. Now [snorts] we feed the the the vector the the embodiment knowledge is already embedded in the model rates. We feed in the video frames and the text conditioning which defines the task and the output essentially is a video of the continuation of these video frames and also um the actual vector that defines

38:13 the direction that the robot needs to move in to actually complete the task. And then you can on the right see how the layer robot um is actually controlled by flux. I mean you’ll have to trust me that it’s actually controlled by flux but you can check that out yourself. Um, and it actually completes exactly the task that it’s uh supposed to. You actually can see the the text bombs there. But um, so that is pretty fun. How that works in the background. Um, let’s take a little look at the architecture. Um, is that we have

38:45 um a video model. Traditionally, it takes certain frames and it feeds them through a variable autoenccoder. um the autoenccoder projects my frames, my pixel space into a latent space which is an abstract representation um of of these exact frames and then we go through this den noising process that gradually reduces noise and tries to predict these next video frames. Um usually when we just predict an image or a video we then have a a decoder as well that then takes these latent these

39:15 abstract representation and brings them back into the pixel space. Um now flux action is a world action model. Um that means that we don’t even send these action vectors um through the viral autoenccoder into the latent space anymore but we just normalize them into the dimensionality of the um of the latent space and we directly dise the uh directly dise essentially the action vector. Um that is a huge example because we essentially dise both in the same transformer. Um in the end

39:47 for the action case we actually don’t even use the um decoder for the um for the frames and video part. We only denormalize the action part. Uh we have the robot actually apply it. Um and that’s how we complete the task. Essentially the video needs encoder and decoder. Actions are already a vector. So we can actually naturally feed them through the through the model that has been mittrained and fine-tuned on exactly solving these tasks. So flux flux to action uh flux 3 action

40:18 um you can check out the research. You can download the rates here. It has dropped literally last night. It’s right now topping the leaderboards and being being state-of-the-art in exactly um this action prediction space. So check it out. It’s pretty exciting research. Um and we’re pretty excited about venturing in this in this domain uh as a new modality. Um and this is also already where I’d like to wrap up. Um what we basically looked at is customizing flux for certain use cases.

40:48 We looked at prompt upsampling and how prompt upsampling can be an incredibly efficient method to actually match user intent with the design requirements into the training distribution of flux to actually get out an image that at scale uh conforms certain design requirements. We also looked at moderation. We looked at sort of static deterministic or more or less deterministic moderation uh that modularly um is embedded in config. Um we also looked at how to embed moderation preferences into model

41:20 weights directly using a using a slider law. And then finally finally we looked at how to um add a new modality to flux specifically looking at flux action and how to tweak flux to actually control a robot and predict the next action. If you want to take away one single thing across all these methods and this is obviously just scratching the surface. If if you want to take away one single thing um about good customizations of AI models in general and also flux specifically it is that great custom

41:50 customizations in all of these cases are as specific as needed and as general as possible. So they solve one specific problem to exactly the extent that it needs to solve them. But it also generalizes to actually maybe solve an entire domain, maybe be a solution that you can also roll out further. So with that, I’ll wrap it up. I have another advertising block here. We are hiring for the product engineers, solution engineers. Um if you want, check that out. You can also find me um find me at

42:20 the conference, find me online. And yeah, thank you very much for attending. [applause] Thank you, Yak. Good job. All right. Okay. Yeah. Let’s give it up once more for Yaku, please. [applause] Thank you guys. I’m so excited for Black Forest Labs. And uh the nice thing about them as well is that they’re based in

42:51 Europe. They’re based in Germany. And they’re the at the front of everything generative media related. and now they do robotics which is so cool. But you know another company that is European based and also at the the frontier of AI is 11 Labs. So our next speaker is actually is going to talk to you about generative identity at scale with orbs and basically uh who who here has used the chat GPT voice mode or 11 labs. So you’ve seen that little circle, that

43:22 orb. If you use that, you probably have used something that our next speaker has built. So please join me in welcoming to the stage creative experience engineer at 11 Labs, Dorian Lods.

44:00 Seems that internet’s not working. That’s a good start. All right. Hi everyone. I’m Dorian. I’m a creative engineer at 11 Labs but currently based in Paris. Um if you want to know more

44:31 about what we do at 11 Labs or if you have any questions later uh please feel free to note the my socials. So I’m here today to talk to you about our how in a creative per perspective things have changed a little bit in the past few years and how how we deal with all these new tools set of tools AI tools enabled and so on. So uh just let’s take uh let’s let me just take a take a step back and

45:03 tell you how I used to make the thing and now how I make the thing that makes the thing which is a bit of a change. So I came from an industry where used to make immersive creative experience websites are mainly based in WebGL or immersive installation or whatnot. They were all designed around a feeling uh refined over thousands of hours. Each one has a specific craft and and purpose. Um [snorts] during that journey um I interacted a lot with uh how you can streamline stuff

45:37 with a system, how you can produce more or just expand your capabilities um with new tools for example comfy UI which is an amazing tool that helped creative new immersive like new deeper uh AI systems but still made for a single outcome and you will see that’s important for And all of this background led me to a specific project I work on which is this guy.

46:07 Um it’s the OpenAI voice the face of OpenAI when they released the voice model. Um few notes but it’s was the first iconic orbs that I had to do. uh everything was handcrafted, had to fit on brand and was the goal was to to find uh a way to visually represent the voice that you talk to uh while representing the brand. So pretty much the same stack than before. It was hand tuned, deterministic, everything you see on

46:38 screen is made with a shader. Um so math actually makes the visuals, all the inputs, all the colors, all the way the thing behaves were carefully tuned. But it was just made for a single purpose. And I spent few weeks designing one sphere. Then I had to design for 40 million voices which is a big change of scale if you ask me. So I joined 11 Labs. Uh for the one who don’t know us, 11 Labs is a

47:10 European firstbased company. Uh we are leading the way on we have our own frontier in the research team dedicated to any voice surface any voice interaction music any sounds to be honest but we do much more than that. We have our own creative platform where you can um dub make new clone your voice prompt another voice um and so on. So many of our or many many creators uh already use the platform. We have the

47:42 web one, we have the app, we also have uh 11 agents now. So we handle our customer support call, we do our own orchestration, many many many things. Um here are some key numbers because everyone loves numbers, right? Uh we support over 90 languages. Uh we already have millions of business developers, creators on the platform. Uh we crossed 600 millions of RR by end of June. uh current valuation is at 11 billion as of

48:13 February and now we actually expanded the team to over 700 people across 50 countries. We have more than 10 million agent on the platform. It’s a lot of numbers I know. Uh but some here are some fun ones. So every day we generate more than 34 years of audio with our speech to text model. Um so the scale factor is definitely here and uh one that uh we we made 25 million tracks with our music our music models

48:43 and different music touch points and one that I really really u that I really love I think it’s my favorite number here is that we so far helped to restore more than 11,000 voices for people who have lost them due to illness or injury. We’re super committed towards that goal. Our impact team is doing an exterior job uh to bring voice capabilities back to the people who actually need it for free. Um but more specifically I joined 11 Labs design. So the design team is

49:14 pretty much all us here. Um we try to push the frontier of both the brand, the product, the way you interact with AI with all our stuff that we have on the platform and how we can streamline that. Um I join with my first uh challenge actually uh was to explore how voices which is our main interaction uh could have a visual identity and how that identity could work across the product. So how you bring more soul visual souls to our voices and how those

49:46 souls should reflect the brand. Uh here’s what we’ve been cooking in the past few months.

50:17 Do not first down. Yeah.

50:58 Hey. Hey.

51:56 >> [music] >> I might have spoiled a few things with

52:27 that video. Uh so um the pitch was simple but yet not so much. We believe that every voice is every face. Um and then we had 40 million voices on the platform. So uh users they can prom voice, they can clone their voice, you have we serve many type of different voices and so they all uh they all represent now more than 40 million voices. And how do you design something

52:58 that is iconic enough that still translate the voice characteristic but keep it in one visual system that each voice needs to bring the identity of 11 labs and it still need to feel like 11 Labs. So it’s all about feeling and developing it at scale. Uh first thing first it needed a reframe. So it could not just be me making the outputs anymore. It was me making the system that makes an output. very close to what we’ve been hearing in

53:28 the past yesterday basically and today this morning uh with code factories and so on but here more on the design side uh the core concept of the or is it’s actually a stack of two layers you have the image layer that carries the brand characteristic the voice that it’s actually linked to the voice and then you have the realtime layer on the top which brings that to life at the interactivity had the living piece to to the one in the back. So, a

54:00 bit of a change compared to the first one I worked on at OpenAI. Uh, and that touch point then is diffuse everywhere across the board. Let’s focus first on the image layer. Um, it’s as it core it’s linked to the voice. So, one voice equals one face. If you make a new voice, you make a new a new or So we need to we went down to the source and we we analyze the voice

54:30 output for air like the audio output for every voice. We extract key metrics, key numbers um that that then fall under different category of pitch, energy, expressiveness that that drive the rest of the pipeline where at this core it’s very much linked to the voice. Uh in the first place we wanted to use embeddings directly but canary became a bit complicated later. Um to work to design the process and to

55:01 really uh expand or to explore how much range do we need to work with. We extracted a set of 200,000 voices the most popular we have on the platform. And we realized for example that they are not distributed evenly. you have much more deep voices mainly for narration purposes. So we really had to focus in those kind of categories where the other ones might fall under like niche where it’s super high super energetic like maybe cartoonish style of voices. But uh that really helped us a

55:32 lot during the process to to explore the thing at scale. Uh first thing first we wanted to um the voices to feel like the brand. So we needed to teach a model to speak 11 laps. Each voice is unique. The goal is to create a generate new voice per new visual new or per voice. So we needed to find a way so a model can actually learn our design language and how that translate. Uh so of the chef

56:02 they don’t really know your brand. Um so we built a synthetic data set to train a specific lura on the top and actually there’s pretty fun thing that we explored is that we steer the inference uh with the caption. So on top of the of the action words we hand selected multiple type of colors and things that we wanted to attribute to different kind of voices. Let’s say high pitch, mid pitch, medium pitch. Um and then we we

56:34 put that in the caption. So during the training um the model could actually steer a bit and like develop a bit more uh the varity and all those keywords were actually were put back into inventory. So at inference level when you rebuild the prompt the model can choose exactly the same keywords like sub keyword that has been in the training data. That was really fun. Um we explore different type of model uh one from our friends.

57:04 So it’s a good segue actually. Um so we we try to yeah we explored different kind of open weight model. The goal was to f to find a good sweet spot between uh between like inferencing cost and development at large scale while still maintaining the quality. um post inference after you generate the base image the idea is that basically you blur it and you crop it inside the inside the base image. So each variation each or has a really really good

57:35 variation into it. Um that’s what we call post inference. Um and that prepares for the rest of the shader. So let’s now focus on the real time layer of those orbs. Um during the design phase we we actually used code as a core design medium. So it lives as a shader. Uh it takes the image in the back and has multiple layers on top of it. Everything was uh made through an

58:05 interface where all the design team was able to tweak and like see how we can feel things. Uh some parameters are tweaked with the sound itself with the audio feedback and so on. Um, WebJ is the interactive experience on the platform on the on the marketing website but uh uh but we converted this to OpenGL support for server server side rendering for many previews or other touch points that cannot actually run the shader themsel.

58:36 Uh the shader itself is composed of basically six six stacks together. You have a realtime fleet simulation that has um the reaper effect that’s linked to the audio. It’s really bring the organicness into it. The goal I might have skipped that but the goal was to anchor the orbs into nature. Nature is the main characteristics of the orbs. You wanted to fill them to feel organic. Then you have multiple phase of of noise on the top to bring it to life. FBM and

59:06 so on. And we have a a further sound wave that is actually a link with the one we had before and the the the brand grain that we put on many of our of our touch points. Um so all of that led to the shader playground like I said before it was the place where most of the craft was happening early on. Um but behind that we actually developed a reviewing tool to debug everything at scale. So one voice equal one output. So you also

59:38 mean that you cannot choose the output and every output had to be right. So we quickly realized that in order to be able to maintain quality across the board, we needed to redo it at scale. Uh and that was really a key factor because you might end up with your debug ones that are super nice and then as soon as you run it at scale, it’s actually fall down. So building internal tools to review things at scale really really make the make a change. Um that’s the less fun part. Uh so I had

1:00:12 an we had an amazing an amazing op process and amazing workflows. Uh and then I run through uh finance. Then they told me that okay D is super nice but for millions times 10 cents it’s way too much. uh so you need to find a way to reduce both latency and the cost. So we went back to the uh to the chart that I shared before where we have all the distribution of voices in the platform and we actually um took this as the core

1:00:45 source. So instead of generating a new image every time, we generated a gigantic atlas at once. And then from this atlas, we actually crop a bit deeper in each of the image. And so uh that help us to not generate every time we we make a new voices but still maintaining a uniqueness. So every voices is unique because we rotate it, we crop it differently, we change a bit to U and so on. uh and they are still really much um locked into the audio

1:01:15 characteristics. That helped to save more than 3.6 million. So pretty good. Uh run it 10 to 15 times faster and have less than three seconds per voice to generate the visual. So it have it’s happening on the back as soon as you as you make a new voice and then it run through all the the process audio process. It spin up it spin up the task on the infra and then generate the one. So less than 3 seconds actually make it not visible for the user. Um and so here’s a recap of the pipeline

1:01:48 for the voices. You have the voice analysis in the first place that maps that map key metrics to uh visual mapping like a medium pen and energy level texture and so on. Then we have the entire generative pipeline I just showed you that prepared the assets, prepared the shader base, um the texture, the preview, the thumbnail, animated MP4, all of these diffused CDN and so on. And at the end you have 40 million plus outputs uh that serve

1:02:19 pretty much everywhere as you can as you’re going to see in a few. Um quality development was really really important. Uh the concept the core design concept was was cool but we wanted it to feel alive everywhere to have to to to follow all the pipelines of the orbs that we have on the platform. Uh here you see on the left you have the the voice cloning um the voice cloning interface. So the orbs change depending on your voice. Uh we

1:02:52 wanted to maintain quality across iOS, Android, uh it’s the one you see in the middle. Every shader had to look the same. Super important for the brand. Uh and we needed to review it at scale. So on the right side, you have like the admin interface where you can review every voice that we have. Um all of that started to grow. Uh we made a dedicated uh service. So we have a dedicated um cloud service for making our orb

1:03:24 visuals. Now it’s called Picasso because it’s painter and and I’m French. Uh it’s a shared API across all our touch points. We we push those orbs everywhere. So in the core app the marketing website iOS and Android app uh 11 music which are which is our music brand branding and expanded into the brand core itself. So we made brand tools motion designs everything now used orbs the carrier brand the carrier of

1:03:54 voice and the carrier of face. Uh it was backward compatible because we had already previous type of orbs in the platform. So we were able to run a back field of 40 million voices. Uh that back that back field run for a few days. Um really stressful moment but it worked at the end. Uh and now they are everywhere. They we cannot hold them anymore. They they are part of our um homepage. Uh the home page the first thing you see on the homepage is the orbs. Um they are part

1:04:27 of the app itself. So you see for example here you have deeper voices has a tendency to be a bit darker. if they have more energy, they will have like blue hints on and so on. So, I really invite you to play with the platform and to try to filter voices um to to see how each characteristic of voice translate into visuals. Uh more renders and they actually came out of the screen which is really really cool and something I’m super proud of. uh we use

1:05:00 them as a core brand uh elements toward everything we do. So on the left here it’s a autoome campaign we made in char uh in France uh where you could have spot orbs based on the on the on the branding of some of our clients. Here is Alan for example or on the right side you have London uh where we took over um one of the main square of London with train line color DNA and so on. So once again

1:05:30 another clients of us and then we the orbs as itself doesn’t carry identity but it’s a shape shifter so we can also use it for expressing um so illustrating the the needs of our clients towards the thing we do. And here another example in the Paris metro. It was really fun to go to work and then see that on the metro. [snorts] Um so here is what we’ve learned uh during this little process. Uh using

1:06:01 machine learning as a core creative medium change how you design. It changed the way you think. You don’t think about only the outputs but you need to think about how you create that. um machine learning AI tools are basically superpowers but they need to be used carefully uh if every design thing that you do don’t overlook it now the cost of making the thing has become so much lower that we have tendency to ship fast and fast and fast and we believe as a design team that the the edge is like

1:06:32 the quality edge is going to make you actually um want more of brand awareness brand quality and so on So uh you need to own the pipeline. You need to own all the outputs like if they were your own. Uh the crash shifted into the data and rules. Uh this is how you trying to maintain all the AI stuff that we have and to be sure that they follow a branding rule, a feeling rule and so on. Uh every stage had to preserve the brand

1:07:03 vision. Like I said, branding is getting more and more key. Um and the patterns revealed what a single output hit means that we need to debug at scale. If you’re making a if you’re making a a pipeline a process that it’s meant to to be deployed 40 million times you need for example to to debug it at scale. So don’t you don’t hesitate to run multiple run of of generation at a smaller scale to be sure that every you can own every single output.

1:07:33 So I still care about every output like I said. Um they are all my they are all I’m all proud of them. There is not any one that I don’t like. Uh but instead of designing them now I designed the condition that produce it and that’s really really important and it’s something that we’ve seen on the code side and we also see it now on the design side. That’s pretty much it for me. Uh thank you for your attention and if you have any questions please don’t hesitate to reach out. uh check out our website or

1:08:03 apps or everything and uh thank you very much. [applause] All right, thank you so much, Dorian. Let’s give it up one more time for Dorian, please. Thank you. [applause] All right. Okay, so show of hands. Who’s here has uh used open weights models before? Okay, cool. So if you did, you probably have used the tools from our next speaker company. So up next is someone from hugging face. He’s going to dive

1:08:36 deep into how a transformers model loads in VLM. So yeah, hold on to your seats guys. That’s going it’s going to go deep. So please join me in welcoming to machine learning engineer at Hogenface, Harry Miller. All right.

1:09:15 Okay. So, hello everyone. I’m Harry and as you just heard I’m a machine learning engineer at HuggingFace and I’m also a maintainer of VLM and in today’s talk I’m going to talk to you about how a hugging face transformers model can be loaded directly into VLM and why because of what we do to it will run just as fast as if the model had been dedicated had a dedicated implementation in VLM itself. But first a little bit of context in case you haven’t heard of VLM before.

1:09:45 Uh, VLM is an open-source LLM inference engine. It came from the UC Berkeley Skylab and is now a PyTorch Foundation project. Um, and the key things that it brings is page detention which allows you to KV cache efficiently. Um, continuous batching which allows you to process a request as soon as it arrives rather than waiting for the next batch to begin. Fast kernels scale out through many different axes of parallelism,

1:10:17 portability by running on any hardware um in any environment and then support broad support for hundreds of models both through dedicated implementations and the transformers modeling back end which you’ll hear more about imminently. And so then the transformers modeling back end um just again more context uh is what lets you take this transformers modeling code from hugging face transformers and run it directly in VLM. So the model definition the config and the weights all come from transformers

1:10:48 and then the runtime and the all the performance critical modules come from VLM. So you kind of get a best of both. And so why might you end up using the transformers modeling back end? Well, there are three main reasons. One of them might be that you’re already using the transformers ecosystem. And so you don’t want to reimplement your model again for each inference engine that you might want to support. So through backends like this one for VLM and then also SG lang um you can take this code

1:11:18 that you wrote once for your transformers training loop and run it straight in VLM or SG lang. uh you might be using it for RL in which case you would be using traditionally uh potentially transformers for your training and then for rollouts you would use VLM uh both with different modeling implementations because previously that was what you had to do. Uh but now with this you can use the same code for training and inference during your RL

1:11:50 loops. uh meaning that there’s no potential drift in the two implementations and the the training should go a little better. And then finally, it might just be the only way that your model is supported in VLM. So today there are 14 architectures which only which are officially supported via the transformers modeling back end. Um and that list is only growing as time goes on. So now we start getting into the into the weeds and talking about how we get

1:12:21 into the transformers modeling back end and what it does. Uh so obviously first we start with starting up a VLM engine. Um and here we have a whole load of uh command line arguments. One of them is to force the transformers modeling back end because for this uh quen 3VL model that we’re looking at uh there is a dedicated BLM implementation. So we have to tell BLLM that transformers is what we want. Um and then we’ve used a a not recommended but possible uh parallelism scheme where we’re using as many as we

1:12:51 can across eight GPUs. We’ve got tensor pipeline data and expert parallel all at once. Um and then finally we’ve enabled torch compile on the multimodel encoder which not all um VLM models will do by default but because of the generality of the transformers modeling back end. If the transformers code uh has a compilable multimodel encoder you can just enable this flag and if your model works it should also work via VLM.

1:13:23 So we first cross the boundary into transformers from VLM by looking into the config where we’re looking at what is this model, what shape is it and then any other features like what quantization scheme is it using and we then once we’ve chosen the transformers modeling back end had a look inside the config we need to choose which transformers modeling backend class to load our transformers model into. Uh and on the right you can see the selection of classes available. Um

1:13:56 and the way that we select them is by looking into the config and establishing which characteristics the model has which will determine the correct class to use. So first they all start with transformers as you can see. Um and then we detect this one is multimodal because it has both a text and a vision config uh in that uh config JSON you saw on the previous slide. It’s because it has more than zero experts

1:14:28 and then forcal lm is the default uh if your model is a causal model or if you don’t specify that you want to do a pooling task with your causal model. And so we choose the transformers multimodal for cos lm class. Uh and what are these classes made of? Because there were quite a lot of them and you’d think that if we didn’t do anything clever, we would end up with lots of duplicated code um and lots of drift and it would be really hard to maintain. Um but all of these are made

1:14:59 of mixins which enable each individual uh capability. So for this one we first start with the MOE mixin which provides uh all of the necessary protocols for expert parallel and expert parallel load balancing. Then we have the multimodal mixin which handles all of the vision encoding the encoding the vision encoding caching um mrop if your model has it which this one does. Um and then merging the vision embeddings into the text embeddings

1:15:30 before passing it over to transformers. Then the causal mixin which is for the lm head and the logics computation and then finally the base class which does most of the heavy lifting and all of the mixins actually now uh inherit this and it ends up at the bottom of the inheritance stack here. Um and that handles the the weight loading, the weight mapping, um all of the modifications that enable this to run quickly um just as fast as a BLM model would. Um and then there is a whole host

1:16:02 of supports this supports that uh protocols for things like quantization, Laura, pipeline parallel, eagle and eagle 3. So now we’re going to look at how the transformers class is instantiated and what we do to it uh during instantiation. So it all looks a bit faint now, but that’s because it has not yet been instantiated. Um first we actually before doing any of that we register two attention shins with

1:16:32 transformers attention registry. Um and that is the VLM attention and VLM MLA attention forward methods. Um then we patch the config so that when the forward pass of the transformers model happens uh one of those shims depending on which is correct for your model is what is actually selected rather than uh flash attention flex attention or or whichever other options the model uh claims that it can use.

1:17:04 Then we decorate uh the language model always for torch compile from the LM’s perspective. Um, and then because of that command line argument I mentioned earlier, we also decorate the vision tower. Um, which tells VLM that you can compile both of these as whole graphs independently of each other before uh before torch compile gets a chance to kind of wing it and figure out on its own. And now we instantiate the model uh using the automodel from config

1:17:36 method on the meta device. Um and for automodel is slightly different to um auto mode for cos lm which might be a class you’re more familiar with if you use the transformers ecosystem. Uh and the auto model just loads what transformers refers to as the base class which is just the language backbone and the vision tower but without the language modeling head.

1:18:10 I might have to step back to the laptop. This appears to have stopped working. Um, then we create the HF to VLLM mapper which is how we specify how checkpoint weights might be translated into how they now look in the Python class that gets created. uh Transformers has a few of these uh built in if they have moved from an old style naive mixture of experts to a fused mixture of experts style. Um so at this stage this is a

1:18:43 pretty minimal mapper but as we move through this process we’ll be contributing to it as we modify the model. So after that we then delete all of the layers that the current pipeline stage is not using. And in this example, we’re on rank zero. So we delete the second half of the layer stack and the final layer norm. And now we begin recursive replace which is where the bulk of the work happens to bring the performance and access to all

1:19:14 the features that BLM has. Um fore models it happens in two passes because thee is a little bit special and there are two different ways which we can do it. Uh for this model it uses the internal routting method which means that we by using torch FX uh detect the structure of the sparsee block. Um and if we determine that it is entirely representable by VLM’s runner um then we replace it wholesale. So we replace the

1:19:44 experts with this MOE runner. Um and that contains the rooted experts and then the routter gets replaced with a gate linear and that gets passed into the runner. Um and that handles everything. And so that lets us do uh things like use monolithic kernels for routing and experts. Uh if there were shared experts in this model, it would allow us to do asynchronous shared expert computation. Um and all of that is managed by thee runner.

1:20:15 And then once that is done, we go to the second pass which is the base class doing a depth first search through all of the children of the base model. So first we walk through the vision model replacing all of the layers that we know how. Um

1:20:55 to its features. So all of VLM’s parallelization is built into its layers. So by using VLM’s linear layers and VLM’s experts layers, that’s how we gain access to tensor parallel expert parallel etc. Then we do the vocab parallel embedding swap um by surgically replacing just the torch nn embeddings module. So if you have a model like Gemma where you do some scaling during your token embedding

1:21:27 um that is all still preserved and we don’t have to special case it on the VLM side. Now the next fuser uh is used which is the QKV fuser which again uses um FX to detect the QKV and output projections um and swap in BLM’s QKV parallel linear and a corresponding row parallel linear um using a so so this is all it’s using torch FX to detect what we need to swap

1:22:00 but then it’s using Python’s as module to actually make the modifications. So we are yet to torch compile anything and all of that is still left up to later stages in BLM. Then similarly we replace the RMS norm which is deceptively difficult for how simple uh it is but we do it because VLM uses um some global torch compile passes to detect where it can fuse layers and

1:22:32 it does it by detecting its own modules. Um and so there are some um RMS norm passes or some compilation passes that the RMS norm is involved in that will only be detected if we’re using VLM’s class. Then we create VLM’s attention instances and as before this gives us access to the rest of VLM. Uh and the key thing that this gives us access to is VLM’s KB cache. Um so we don’t need to use any KB caching that has been implemented on

1:23:03 transformers. Um, and VLM gets to do all of the clever stuff that it can do with the KB cache like offloading, um, disagregated prefill between nodes, um, KB cache reuse or all of that fun stuff. Then anything that we haven’t touched yet, we make sure that it is on the device. Uh, so the only thing that changed there was those two norms in the vision model. And then lastly, we just make sure that anything that Transformers insists should be in FP32

1:23:34 stays in FP32 even if the rest of the model is FP8. So we’ve built the model um and now a request has been received. We have had it scheduled through VLN and we’ve reached the base classes forward method. Um and this is the first hop back into transformers. Uh so we simply call the forward method on the original Quen 3VLOE model class

1:24:04 um and pass in the input ids or the input embeds um in this case it’s the inputs embeds because it’s a multimodal model and VLM pre-mbbeds the vision part uh of the prompt and inserts it into the text embeddings and passes that all in one go rather than relying on the model itself to to do that we explicitly disable transformers is um KV cache because VM owns it through those attention modules. Um and

1:24:35 then we just forward through the position ids with a slight modification for MRO models um which happens in the multimodal mixin for this particular type of model. And so while we’re going through the forward pass of the transformers model, there are two seams where we return back to VLM um in kind of performance critical ways. Uh and so the first and most interesting one is the attention module. Um so on the transformers side,

1:25:08 we’re starting again in the the transformers native class. It goes through its registry to choose which attention method to use which we pre-registered earlier as the VLM um attention shim and then selected it by modifying the config and then we enter that and transition back to VLM where VLM does some minor shape manipulation um gets the attention kernel to use which is any attention kernel supported

1:25:38 by your device for this particular model in VLM it gets the KV cache using uh the attention context and then it actually calls the attention and passes the result back to transformers to continue with the rest of the model and the second is thee through internal routting. Um so as I mentioned thee has a special case where if we’re able to detect an entirely representable sparsee block then we can replace it entirely

1:26:11 and actually the forward method that we use is just experts forward and everything happens inside that runner class. So we go straight into the VLM side. We gate the hidden states. We get the top K ids and logits. Perform the rooted experts for computation and pass the results back to transformers. And after this point the transformers model is done. It has provided all of

1:26:43 the information about what the graph of the model is and all the computation for well and all the journey through the computation rather because as we saw the modules are all VLM modules um and then after that the logic computation and LM head evaluation is all done by VLM um and the sampling is also handled by VLM. So there are a few features that we haven’t used with this model that I want to talk about. The first is external

1:27:14 routting. Uh so if your block isn’t the same as something that we’ve anticipated, uh that doesn’t mean that it won’t work. It just means that we will only replace the experts themselves. Um so that gives you the crucial performance because that’s where most of the work is done. Um, but it gives you the flexibility to do whatever you might come up with in the routing or anything around it. And there’s also shared experts, which I

1:27:45 alluded to earlier. If they had existed and they were detected in the internal routing, they would also have been included in the MOE runner. And depending on your configuration of BLM, um, it would be computed in a way that is more efficient than could be done with external routting. Then there’s Laura and Eagle and Eagle 3, which we just didn’t touch on because these are kind of determined by the checkpoint you use and then the requests that you pass.

1:28:17 Then there’s a a whole load of different attention flavors. So there’s encoder only attention which would be used by models like BERT or sliding window attention that you might find in Gemma models. Um and for encoder we support encoder only models um not no encoder decoder models but for sliding window attention it can be interled um similar to how they started doing it with Gemma where you would alternate between full and sliding attention. Um and then also

1:28:49 MLA attention which comes through a separate um fuser and that actually gives us the KB cache compression benefits that we didn’t used to get when we used the we just used to pad to full attention when you loaded a a MLA attention model. Um and then finally pooling and classification tasks um which would have been automatically used had we used a uh a pooling model. Um or we could have

1:29:20 manually selected it and BLM would have created a classification head using the uh the LM head or it could have given you back the the raw embeddings out the end of the model if that is what you were after. And so to recap, we looked into the config, determined which hugging face architecture it was and then which transformers modeling backend class we should use. Um we loaded it into this generic class in BLM

1:29:52 and we manipulated it such that all of the advanced features of VLM became accessible. Um and all of the performance that comes with them came with them. Um then we went through the forward pass of the base model with two important seams um where we go back into VLM and do some non-trivial um work and then we pass back to VLM to do the logic computation and the sampling

1:30:23 and that was my talk and the slides are available on the QR code there um if anyone wants to look at them again Thank you. [applause]

1:30:59 Sorry. That’s what happens. All right, I think we’re on a break now. Thank you so much everybody. So, I think we were going to take a little break and we’re back in half an hour, I believe. All right, thank you so much.

2:18:56 Hello, welcome back. All right, let’s get it for you guys. Okay. So um so far we have spoken about you know the AI economy like we talked about harnesses, software factories, we talked about generative media and all those tools are for you guys to build agents to build applications that use AI. Who who here built actually with AI? Okay. Yeah, most of you guys do, right?

2:19:27 And the purpose of that is to create value to for people, right? Whatever agent you’re building, whatever tool you’re making, ultimately is to bring value to your users. Our next speaker is actually deep into uh the the the data and um she works at Stripe and what they see is that the entire economy is changing thanks to AI and uh she’s going to come to to tell us about and give us

2:19:57 some insights about what Stripe uh what stripes data is revealing about AI. So without further ado, please join me in welcoming to the stage the head of product of AI and startups, Ariel Lubai.

2:20:28 Hi everyone. Thank you. Thank you so much for being here. Um, I’m super happy to be here and I hope I will get some slides at some point on the screen. [laughter] Uh, can I get the slides in the please? Okay, perfect. Okay, let’s come back there. Okay, perfect. So yeah uh really

2:20:59 happy to be here. Uh so at Stripe we work with the fastest AI growing companies. We we have this luck and many of them are actually in this room. So really happy to be here and we have a front seat of what’s working, what’s not and like how the best companies scale today in this new world of AI. So today I want to share some of those insights with you guys and you know have some thoughts on the playbook because this has been completely rewritten in the

2:21:31 past few years and even months. So first like let’s start with uh figures sorry um first we are working with um 1.6% six% uh of uh of the GDP globally that is processed by Stripe and that is really something that puts us in that front row seat at the five patterns that we’re seeing. When the top AI companies build faster, I think this is something we’re all feeling. Two, they sell

2:22:02 globally by default. When you’re building today, your day one market is now the whole world. Um, three, pricing is evolving completely. Like there are a lot of workshops on pricing today by the way because how you price has completely changed in the past few years. Um, and four they adapt their go to market motions as well. We have like channel sales that are becoming increasingly important. Uh, we’ll dig into that afterwards. And five, they monitor fraud

2:22:33 early and often. Why? is like what’s creating potentially viral growth with agents is also creating just a whole new ground for fraud attacks for your business. So we’ll dig into it of course but first let’s look at like big picture what’s happening um in the world on that. Um we saw the top AI companies grow by 145% in 2025 which was already impressive but like in 26 this number

2:23:05 grew by 195%. So tripling in a single year. Uh that growth isn’t slowing down it’s actually accelerating today. And the top ones that you know are even more mind-blowing of course like lovable they reported 100 million in eight months. uh eight months later they were actually at 400 million. Anthropic that you all know started in January 23 with zero and went to um to 1

2:23:36 billion in in just two years and now they’re at 30 billion. So that’s just completely mind-blowing. And this isn’t just B2B like we saw this in in our link data like consumer adoption is also increasing a lot. like it doubled just from 6 million to over 14 million just in one year. Consumers are also spending more. What’s actually interesting is that they’re spending more on AI tools. Like today, we’re around like $360

2:24:09 uh per person per month on AI, which is basically double what it was three months ago. It was around $180. And this is basically like basically what the average European spends for like phone service um uh internet and streaming services combined. Like people aren’t treating AI today like some streaming service that you can cancel uh but they’re treating that like a critical utility that they absolutely need.

2:24:41 So what’s behind all this? We’re going to deep dive into these different patterns starting with speed. So just like you no longer need months of engineering to uh you know v code and accept the payments now you no longer uh need month of engineering to ship software. So what’s actually interesting is that this chart shows when it clicked. Like if you take a look at iOS app releases, they were actually declining at the end of uh 24 and then

2:25:13 the agent codic tools hit. And so this is where we saw the iOS app launches just grow again and not like small grow by 24% month over month. And so now even like now that we embed payments into developer platforms like Lovable, Riplet, Versal, etc. Um it’s actually even faster to go from your idea to first charge like even faster.

2:25:45 So what you can see here is that it’s actually taking less than six weeks from first idea and like sandbox to your first charge. Like only six weeks. That’s really crazy. And that strategy is faster for technical folks. So I I’ve played a bit with a prompt to show you how it looks like today to build with cloud and stripe. Uh because we’ve we’ve made a lot of work to make sure that we have an agentic developer experience that

2:26:17 enables you to build without having uh to leave where you’re building and having to go on Stripe dashboard and and build stuff. So um let’s say you are a developer today and that you’ve built an app and now you’re ready to set to set up payments. So let’s jump into cloud and see what it looks like. So I have my prompt here where I’ve asked you know to set up payments. I want um invoicing and subscription and so

2:26:48 yeah. Okay. So here you can see that Claude correctly recommends Stripe which is of course what we want and that they will be more powerful because they are connecting to Stripe MCP and the skills that we created with this. So Claude uses the Stripe integration recommener um MCP tool along with those skills uh to know what to use and what to ask to the user. So it’s basically asking like where are your customers located? The

2:27:18 reason it’s asking is because like cloud knows now that they need to have this answer to provide the best recommendation for the integration. Um what’s interesting is that they recommend checkout which is a stripe integration and not like cloud element for example because it knows that um for for a better subscription integration we need checkout. So that’s again like showing that uh it knows the right thing.

2:27:48 So then I responded like my customers are split between the US and the EU uh with a particular concentration in Ireland. So what’s interesting now is that I get feedback on the type of tax knowledge I need to have. So basically uh I need to know that digital subscriptions um are taxable here. And it’s actually interesting because it it recommends me to use stripe tax um because I don’t want to have to know all the regional tax rules for all the

2:28:18 countries that I’m going to launch to. And so Stripe Tax is going to help me without uh without having to know those rules. Um I can see that it’s provisioning a Stripe sandbox for me. Um, what’s actually interesting is that you don’t have to, you know, go outside of the flow here to to get a sandbox. Um, the agent uses the new CLI command to create the sandbox without needing any stripe credentials, which was not the case

2:28:49 before. Ed the command also fetches the API keys and securely stores them. And that way I basically don’t have to break my flow and I can build my integration like without having to to go away. And so with the key sorted I can see that now Claude is checking all the different skills um that we stripe provide to make sure that they’re um it’s building the right thing. So, it’s saying that it has everything

2:29:20 it needs and it’s asking me like just do you want to proceed with MCP Stripe create product and and if I want to continue with that and create my checkout. So, I’m pressing yes here and we’re going to fast forward and try to see the checkout. Um, so this is what it looks like. Um I’m going to test if the checkout works properly and so yeah that’s how uh it would look like.

2:29:51 Okay so it’s done like strap and claude are basically making it easy for you to for you and like all the developers to build set up payments and start monetizing without going away uh from your uh building interface. So what does this mean for you? Like many of you are probably already doing this but I want to reemphasize that you know in the playbook today it’s really important to obsess over developer productivity like the best

2:30:23 companies today are really working on speed and just making sure that they completely collapse build times. Second like build, sell and iterate at the same time. the linear playbook that we had before like you build your product then you go find your customers isn’t just working anymore because of of the fact that too many people are building at the same time. So you basically need to do all three at the same time um in parallel.

2:30:53 And then on the build build versus buy question, we’re actually quite convinced from what we’re seeing that the best companies do both like they built what’s making them differentiated and they actually also buy you know the infrastructure and building blocks that they need for their products. So two like getting to market is one thing but it is also a big question of like where am I going to sell and how am

2:31:23 I going to get those customers abroad. So the data confirms it completely. Um a few years ago the fastest growing SAS companies were basically reaching 25 countries in year one and 51 by year three. Um and today the top AI companies they’re already at 42 countries in year one and at 120 countries in year three. So this is like what I was saying about like day one market is basically the

2:31:54 whole world like it’s not about having a presence in in those countries. It’s actually getting revenue uh from these countries from day one. So Gamma is a great example. you probably know them. Like they’re based in SF. It’s an AI powered like slides and website builder. Um and they reported 100 million in revenue in year um in year one and the majority of their revenue comes from outside the US. So that’s definitely showing how um the the

2:32:27 trend is going. And today if you take a look at overall like all the top AI companies 48% of their revenue comes from outside their home market. Like nearly half of a dollar that you get or a euro is actually from a foreign country. And this figure was actually 33% um a few months ago. So let’s take a look at which markets are actually spending a lot on AI tools.

2:32:58 Um so not surprisingly we can see that the countries with the high GDP growth are here. So you can see France, the UK, South Korea. Um but we also have like new emerging uh countries that are spending a lot on AI. So for example, Switzerland, Poland, Turkey and the demand is there. So now um as you’re building your company, you need to think about like how am I going to get that demand? how am I going to get those customers?

2:33:29 And like what we’re saying what we’re seeing is that you really need to meet buyers expectations and meet them where they are. So the way we’re doing this is that you offer local currencies and local payment methods. That sounds easy and that is actually easy uh to do with St. But like basically it’s really meeting them where they are and with their habits. So localized pricing for example its own sounds like completely simple but basically it increases 18%

2:34:02 higher crossber revenue like it’s crazy the impact that it has. Uh and basically when you add one just one local payment method for a country you have at least 7% conversion uplift uh in this country. So, three checks to pressure test your global readiness as an AI company today. First, you localize your prices and your payment methods. Two, automate tax collection because like no one wants to get in trouble

2:34:34 either. And three, track performance by country like performance um conversion and obsess over optimizing those. So now pricing um because the old models are are not working anymore. Uh now we have to think about like new pricing models. The interesting thing is that like value is elastic. You probably know that if you’ve taken a look a bit at pricing and so everyone might be using the same AI

2:35:05 tool, but they might not using the same way. Um, so take two users. One is an engineer. They’re going to bed. Next day they’re having a code review. Um, and the output is really engineering output. The other user is just an everyday user. They replace uh Safari but GPT and the output is just an upgraded search experience. So for this like the cost is not the same, the value is not the same and so

2:35:36 the pricing should definitely not be the same. So this change in value isn’t new. We had this with all the technology shifts that happened like for example we had local host hosting at a time. So it was normal to download the software for a yearly license. Then cloud computing arrived and so we had auto updates of softwares. So it was actually uh normal to move to subscriptions and today like what do we have with AI

2:36:09 so that’s basically where you’d expect me to tell you what the the the best answer is. Um so good news bad news bad news first we it’s still early like we don’t have a gold standard of what all companies are doing and it is absolutely the best but good news is that we’re clearly seeing um emerging a pattern um take for example poke to our MC today

2:36:40 um a developer tools company that you probably know that’s been around for around a decade basically and when AI tools arrived they pivoted their product they started adding AI um AI tools AI features in the product and they also started to adapt the pricing like layering credits etc. So they started with flat subscriptions which is how you stick and create the relationship with the user and then added credits as AI

2:37:12 became core to the value of the product. So the result is that today they’re targeting 1 billion ARR uh this year and they’re not the only ones to do this. This is just one of the examples of what we can see. So today we have like two out of three companies of the Forbes AI50 um listing that are using usage based pricing. So a majority of these companies are actually using a hybrid

2:37:43 model. They’re using subscriptions still for the the stickiness of the relationship, but they’re adding a lot of usage based pricing to make sure that the value is actually reflected in the pricing. So it’s good to know that you have to do some form of usage based pricing but getting it right is actually another story. So in our view there are three things you need to do to add uh to your playbook for pricing.

2:38:14 First price in your customers language like developer think in tokens enterprises most of the time think in seats like you have to think about these uh like who are your customers and make sure that you match their pricing with the way that they experience value in your product. Second, you want to show like as close as real-time visibility on what they’re consuming because like you know this is this is not a nice to have and this is

2:38:45 basically how you prevent churn as a company. Like this means finding ways to make sure they’re seeing real time what they’re consuming. And three, we really see that it’s it’s strong to sell credits and not costs because like at one point your customers will exchange money for credits, but then the way they will interact with your products will not be with like each euro or each cent. It will be with credits and it will be related to value mostly, not costs.

2:39:18 So now we’re going into go to market motions changing uh which is also a big part even if like a lot of people are actually taking that bit of the side um but that’s actually a big motion changing as well. Um actually entirely new motions are emerging. You might have seen that and the biggest one is channel sales like it means where your product is going to be to be sold and how. And like today in the AI go to market motion

2:39:49 for example a lot of um companies are actually buying open AI or anthropic via their cloud provider. So you need to think about these different channel sales that are arriving and a lot of AI companies are actually building their own marketplaces where you are going to be able to be a reference and to be to be bought as well. And of course like the billion or maybe trillion dollar question like how do you sell your products to your newest buyer

2:40:21 which is you know the agent like we’ve spent decades thinking about what is the best way you know human psychology I will price my product 999 and not€10 but like agents absolutely don’t care about any of this and today I think this chart is one of the clearest signal of like I need to think about my agent as a buyer because like for example this is trade docks uh so our public documentation the pink

2:40:52 line is uh how it’s read by agent and the purple line is by humans and as you can see like this very summer in 26 we had the two lines crossed which means that our public documentation today is more read by agents than by humans. So we we live it every day at ripe like we have to think about okay how is my product how is my documentation read and understood by agents. So you have to

2:41:22 change the way you you do things. You have to change the way you are going to showcase your products and make them discoverable by agents. And go to market is like the tip of the iceberg because if you think about like changing uh channel motions then you also think about like adapting your product to that making sure that your onboarding is going to reflect that new uh onboarding motion. Pricing and revenue of course is going to have to reflect that as well your organization.

2:41:54 Um so for example if you have enterprise sales you are going to have to adapt this to the new uh new go to mac and motions as well. So this is how I would think about the playbook. Like basically today the products are channel agnostic and the customers themselves are channel agnostic. They don’t really care about where they buy. So they may be self-s served at some at some point graduate to enterprise and then like have an agent implement a new product. So you need to

2:42:25 think about all of this journey uh for your product. So have a clear graduation project pro progress from productled growth to salesled growth. Uh then make sure that you have unified system you have like one product catalog that you have one customer object um one data model and that you don’t go away in all directions. And third, like as I was saying, please design for agent readiness. Like make

2:42:56 sure that your product is ready to be discovered and bought by agents without a human in the loop. Okay. So last pattern, fraud. Um this is actually something very important that you need to think about. Monitor fraud um and risk like early and often. So what makes it hard basically is that like it doesn’t show up in one place like at all at all time of your product life cycle they’re going to be a potential for an attack. So if you think

2:43:29 about account creation, payment method added, payment processing and then after that like post usage all of these places are potential um places for fraud. So let’s take first like sign up top of the funnel. The thing is, you’re going to have someone that like a bad actor that signs up again and again, and so they’re basically just stealing all of your user credits um at at once. One in six signups, to give you an idea,

2:44:01 today uh involves multi- accounts abuse. So, they’re basically farming you free trial um without your evering a dollar from them. And the problem also is that JI today has made it very hard to, you know, find fake accounts because they actually look real. And what’s interesting is that multi- account abuse is often like exactly how free trial um abuse happens at scale. So

2:44:32 you can see that it increases dramatically like this is how it accelerated in the straight network. So you can see that in 2026 we have this crazy increase of free trial views because like if you’re having success like you’re actually also an attractive target on the internet. Um what’s good news though is that it’s a solvable problem. Like just as you can see an exponential growth of the attacks, you can also see a very strong

2:45:04 and steep decline of the attacks if you have the right uh solution. Like just in four months we blocked like three million attacks like this for free trial. So that’s definitely part of top of mind for us. And then like of course there’s another vector you you might be familiar with pay as you go pricing. So pay as you go abuse goes with that. So what happens is that a consumer would basically consume let’s say a thou a

2:45:36 thousand or thousands of dollars of tokens in a month and then at the end you try to get the money and the card is just invalid. So it’s kind of like you know Michelin start like we say in French uh but for tokens right. So by the time you know the compute is gone and you can’t charge anything to that user. So pay as you go is an amazing tool for pricing but you have to think about how to manage fraud in that case. So the companies handling

2:46:06 that well are monitoring usage signals in real time and make sure that they see all the different signals uh and and avoid that from happening. Okay. So what do you do about this like three things again? uh please monitor this at every stage of the life cycle from account creation to post usage. Um so defend all of this for your customers. Then I would recommend to close the gap between usage and billing. Like you don’t want to wait a

2:46:38 month to bill. You want to make sure that you close that gap as much as possible. And then like please build fraud into your launch plan. like it’s not necessarily the best or more most exciting part about building but that’s actually incredibly important like at every step of your product you need to think about okay if I am a back actor now and that I want to take advantage what does the attack looks like and so in my view if you haven’t thought about this or you don’t have the answer yet

2:47:09 like you’re not ready to ship it so the playbook is being rewritten We all know that. But like how you build, how you go global, how you price, how you adapt your go to market motions and how you manage fraud, that’s what’s changed. Not just the tools, but actually the whole starting line of building a company. And you you know that you can build in hours, you can be present in 42 countries on day one. As we can see, you

2:47:41 can price for value. Um, so that’s actually in our view, you know, the funniest and maybe craziest time of like building uh for for building a company in the world. Um, so we we hope that these insights give you some some idea of like how to build this and yeah, like our job at Stripe is to be there with you to help you scale. So um I’m really excited to see what you build. Uh and just please know that we are there if you need to scale as well. and drop me a

2:48:12 message if you want to chat more. Thank you. [applause] >> All right. Thank you, Ariel. Let’s give it up for Ariel once again, please. [applause] All right. Okay. I think what uh the fact that um or the information the piece of information that uh stayed with me through this presentation is that you need to design for humans but you also need to design your products for agents because they might you know be purchasing your products as well which is I think is awesome. Um I asked

2:48:45 earlier a question. I said how many people are builders here? And many of you raised their hands, right? Like yeah, most of you are builders, right? Like you’re building products. Um and you’re uh maybe chatting with agents. Um my personal experience is that I kind of stopped uh typing to my agent. Um so I only chat mostly with voice and I find it like a a more natural way to interact with agent. I read faster than I than I hear. But I also I I talk a lot faster than I than I than I type. Uh I don’t

2:49:17 know if anybody shares that experience here. Who talks to their agent? Yeah. Okay. So it’s a actually a good number. Well, our next speaker is here to talk to us actually about voice AI and real time. Um so without further ado, please join me in welcoming to the stage chief scientist and co-founder of Piano AI, Erve Ba.

2:50:01 Okay, let me try this thing. Yeah, it’s working. Um, hi everyone. Uh, thank you Rau for the introduction. Uh uh it’s the first time I’ve been ever announced on a stage by an MC. I feel like a bit like Celindio in Paris uh tonight uh or today at least. Don’t worry, I won’t sing anything but I will talk to you about Voice AI and in particular about uh speaker dization. Um so uh quick

2:50:32 introduction about myself. So I’m the co-founder and chief science officer at Pianoai. I’ve been an academic researcher most of my career up until two years ago when we uh created this company and um so yeah I’ve been mostly focusing on speakerization for the last 15 years and uh it looks like people think that uh uh I know a few things about speakerization and I hope today that uh I’ll tell you what speakerization here is and that you’ll

2:51:02 learn a bit about how that works and uh yeah that at least you you take something out of this talk. So what is speakerization? So speakerization u basically uh is the task of taking a recording of a conversation between multiple speakers uh and outputting some metadata about it. What you can do starting from a conversation is basically do voice activity detection first where you would uh detect uh the parts where uh someone

2:51:33 is speaking and when nobody’s speaking. That’s voice activity detection. The second step that you can do on top of that is uh splitting those uh very long uh speech turn regions into uh single speaker regions. Uh that’s this task is called segmentation. So it includes both uh speaker change detection as well as overlapping speech detection. So in this example here uh in in the middle you see that there are it looks like there might be uh u at least two speakers because uh

2:52:03 there’s a uh someone interrupting um uh another one or starting speaking after the end of the another speech turn and a bit later in the conversation there is a very small speech turn that overlap with the other one. This these are usually what we call back channels like mhm okay yes that you say when you are nodding at someone uh uh talking to you and those events are very important to uh uh for voice AI agents to uh basically understand really the core of the conversation the words are usually not enough knowing that when someone is

2:52:35 interrupting someone else or nodding at what they are saying it gives a lot more information about the the conversation and the next step really what we call speakerization is then once the the conversation is segmented into speech turn is to actually assign each speech turn to the right speaker and this is done without knowing in advance the number of speakers. So that makes the task more difficult than usual uh supervised machine learning task and we don’t know the identity as well of the speaker. We have to uh basically

2:53:06 generalize to any speaker even those that are not part of any training sets. So that’s what makes speakerization difficult and so there are many tasks uh where speakerization can be useful uh and lots of those task can fall actually under the knowing who said what is just as important as what was said. So for instance uh for video dubbing application when you want to start from a a video in French for instance and you

2:53:36 want uh basically to put it on the internet and reach a larger audience an international audience uh then you can do use video dubbing uh tools to uh basically translate from French to other languages and making sure that the you know the conversation remains natural. So you have to put the right uh uh speaker clone uh voice to on the right speaker and for that you need to know exactly who speaks when in the original audio. Other task uh obvious task are meeting note takers uh especially in

2:54:07 hybrid meetings where uh some people are joining from the same meeting room. when there are multiple streams, speakerization not necessarily uh is useful, but when people are in the same room, that’s where it’s it’s really useful. And this allows to to make some nice uh uh summaries of the conversation of who’s supposed to do what after the meeting, for instance. And other applications like podcast intelligence where you want to track one particular speaker in multiple podcasts or track the host of a podcast across episodes.

2:54:38 And um so we have this uh open source toolkit called uh piano to audio uh that uh allows you actually to get from raw audio recording to uh uh an actual speaker deration. So it’s it’s become very popular over the years. Uh I have some vanity metrics here uh uh thanks to our friends at hugging face. It’s also very visible there. Uh it has uh on average 10 million downloads every every month. And basically in a few lines of code you can get from the audio to the

2:55:09 actual speakerization. So here in this example we start for the first block of the code as basically downloading uh the community one model the open source community one model from hugging phase. You get this uh object called community one here in the P Python code and then you simply apply uh this uh uh pipeline on a recording of a conversation and you get the prediction that you can then iterate over to know exactly start time, end time and the actual speaker time and this really uh in in three lines of

2:55:39 code. Um yeah so this uh that I described was batch in the sense that uh the whole processing happened after the conversation took place. So why uh go real time? So there are many uh reasons why we we would want to go real time. Uh for voice agent it’s not that obvious because usually there’s one person talking on an or to an agent and uh you you don’t really care about uh speakerization because there’s only one speaker. But very soon

2:56:11 and it’s already there actually the agent will listen to to the conversation while we’re having a meeting for instance and to a use case would be to flag some speakers interrupting another speaker too often or to flag a speaker talking too much. Uh and doing that in real time is is necessary. And uh the next step is also that we soon are going to have robots in our house. And uh to basically answer to uh a child or to the adult in the in the household, you would uh basically need

2:56:43 to know who’s who. And this is really where uh real time voice darization speakerization is useful. So now that we know that it’s actually useful, how uh did we at PenAI uh manage to go real time? Uh I need first to take a step back to explain how the batch speakerization that I just described earlier in three lines of code called community one works internally. So uh I’m going to spend some time on this slide a bit more. So um basically

2:57:15 given a let’s say a 1 hour conversation uh a recording of a 1 hour conversation how it works how community one works today is we start by splitting the conversation into pieces let’s say 20 seconds or 30 seconds pieces. So each uh green rectangle here in in this slide uh actually uh represent 30 seconds of speech let’s say you see that they are overlapping a bit. So we we move that in a in a sliding uh manner uh uh in a sliding window manner and then we apply the first uh neural

2:57:46 network that basically performs uh speakerization on each chunk separately. So why do we do that? Why why do we uh process the conversation pieces by pieces rather than uh trying to do that globally? It’s because uh working on chunks allows to uh have a limited number of speakers as output. So the the the machine learning task is actually easier and safer to to solve like that. So basically the model is trained to take 30 seconds of audio as input and returns uh basically a matrix of let’s

2:58:18 say uh the time and the number of speakers uh and um it will output zero and once whether each speaker is active over time. And the problem with this kind of a first approach is that at that point uh the same speaker on the first chunk might have a different uh label on the same for the in a different chunk. So we need to rely on a second step uh where each speaker in each chunk here represented by symbols the triangles the circles and the stars are projected into

2:58:51 a highdimensional space using speaker embeddings. I’m pretty sure you’re all familiar with what embeddings are, but basically those are uh highdimensional representation of a thing. Here the thing is a speech turn. So each speech turn is uh projected in into the space. Uh here it’s represented as 3D but in practice it’s uh Oops. Uh in practice it’s um it’s coming back. Is it? It is. I should probably stay here. Um and uh this

2:59:25 speakering model, it’s a neural network as well. It’s trained to have two uh speech turns of the same speaker close to each other and two speech turns from two different speakers far away from each other. And this way we can actually find uh clusters in this space. And this is how basically we managed to reconcile all the uh the chunks finally and be able to basically stitch uh uh all those small speech turn together and and get the final results. So that’s the starting point of um um of speaker

2:59:58 dization in a batch manner. So the question is how did we get from uh this uh batch diorization to an actual uh realtime implementation. So the first thing that we uh worked on is basically the clustering part that I just mentioned and instead of having it uh global having it incremental. So uh this uh um animation will will say it all. So uh so basically starting with a conversation uh between two speakers here in batch dorization we start by

3:00:31 extracting speaker embeddings from all those speech turns. So you get them all those points are actually speaker embeddings and you simply do uh clustering uh into that. But if you turn to streaming you don’t have access to the future because the conversation did not happen yet. So the idea is that uh the clustering algorithm needs to work incrementally and that makes its job way harder because you only have a limited view of the conversation. So uh you see here uh this is a fake example but it takes some time for the clustering

3:01:01 algorithm to realize that there is actually two speakers and that’s why it makes um you know switching from batch to real time quite difficult if we if we were to stay with the you know chunk wise approach that I just mentioned. it still managed in the end uh to find the right clustering but you saw that during uh the actual uh conversation it made some mistakes and so with this kind of approach uh this is actually implemented in the dieart open source toolkit which is uh a

3:01:34 creation of our CTO Juan Manuel Ka and we managed to uh basically get slightly uh higher diorization error rate uh at around 5-second latency because the chunks are around 5 seconds long or at least the steps between the chunks are 5 seconds. So we said okay uh that’s great but the next step would be to manage to lower this diation error rate at least to be as good as a batch speaker derization and also this 5-second latency might be

3:02:05 a bit too much. So um what we worked on next is basically uh turning this uh chunk wise segmentation into something a bit more streaming that we call causal segmentation where instead of waiting for the whole chunk to be available to process it basically pro processing in in streaming. So that that brings uh a lot more uh of other difficulties. So uh let’s focus for instance on speaker chain detection uh in batch mode. uh what we you we

3:02:35 usually have access to the whole conversation. So speaker chain detection can be done like that. You slide a window uh around the course of the conversation. You compare what happens to the left of the the segment and to the right and you compute the difference and you get some nice peak where there is a speaker change. But in streaming uh the problem is that you don’t have access to the future. So it takes some time for the model to actually reach uh you know confidence enough uh to um enough confidence to basically take a decision and this uh small amount of

3:03:05 time uh that it needs in the future to to decide whether there was a speaker change or not. We call that the look ahead and that’s basically a a a latency that we can’t get rid of because if we at one particular point of time in the conversation you need to decide whether there is a speaker change uh unless you have a very strong modeling of the pro or something like that. You really have to wait whether someone else is starting speaking to to actually take the decision. So that brings this uh

3:03:37 incompressible uh latency that is added to the to the thing. And uh so we we went ahead and uh and implemented that and we uh with with basically lots of research on the on the research team on this particular segmentation streaming model uh we managed to actually using much larger model than the chunk wise one and also some uh some uh implementation of loss function. I won’t go into the detail.

3:04:07 I’m happy to discuss that after that. But we managed to basically reach the same level of accuracy as the batch speaker deration with a a very low latency of 400 milliseconds. And this 400 milliseconds actually um involves many things. It it involves also uh round trip from uh uh the client to the servers and back. So I’d like to spend some time on the actual 40 400 millconds latency. So what’s in those 400 millisecond latency?

3:04:38 So um we start by a client uh having a conversation locally. The the first latency that you get is basically that the conversation is split into 100 millisecond blocks. So you already have to wait 100 millisecond. Uh let’s say at time t you have to wait 100 milliseconds to build this block. So already at time t we already have 100 millcond latency. This latency that I put here uh is part of the 400 millisecond latency and it’s not something that is often uh counted

3:05:10 in the numbers that you can see here and there. But it’s actually also something uh that is important to take into account. Then uh the next step basically is we are relying on the Cloudflare um durable object uh uh technology to basically send this block size to the uh closest uh Cloudflare server. So this on average takes 20 millisecond but it really depends on where the client is and uh how good cloudflare is basically. Um and

3:05:40 on this server what we do is we receive many streams and we batch them. So we batch lots of 100 millisecond blocks so that uh we can process multiple streams in parallel. So this is very fast and then once we’ve uh built those batches we send them to model where uh the GPU horsework lives. So basically for segmenting those 100 millisecond latency uh a batch size let’s say of 32 32 streams let’s say that’s just an example here but it takes

3:06:11 around uh 50 milliseconds just for the inference and also you have to take into account this 200 milliseconds look ahead that I mentioned earlier. So basically when you reach time t what the model the neural network does it predicts what happened 200 milliseconds in the past. So this is uh this adds up and then in parallel uh we we find this trick of actually running the embedding extraction and the clustering in a totally async manner. So this does not delay the prediction whatsoever. So

3:06:42 that’s why I I put it uh zero millisecond here and then we do the the run trip back uh so five more milliseconds here. Then uh we basically uh using the state that is stored in the in the cloudfare we basically stitch the pieces together and ser serialize it to send it to the client. So in total you get those f four u uh 400 uh milliseconds latency and if we go into more details so

3:07:12 basically there are just not just one latency basically there are many latencies here. There is the network latency that basically really depends on the client how close uh the client is to the networks how fast their connection is. So that accounts for like 50 millconds on average but this can be moving parts. Uh then there’s the um uh inference latency that I mentioned the 50 millisecond. So same uh this inference latency you can tweak it a

3:07:44 bit. Obviously if you make a a higher batch a larger batch it will take more time but then uh your cost will be lower because you can process more [snorts] streams uh with the same u with the same GPU but if you want to be faster then you have to uh lower uh the the b the batch size and uh uh your cost will go up. So really this inference latency this is where you control a cost uh latency trade-off and then there is this algorithmic latency that takes basically

3:08:15 most of the uh uh the latency here and same uh it controls a new trade-off. This is a trade-off between accuracy and latency. The more you can uh wait in this look ahead the better the prediction you can make. So this is really uh like a as I say a a trade-off between the two and an easy way for to to to reduce the latency would be to reduce the block size obviously uh here it’s 100 millisecond but that has repercussion on basically the inference latency because you get way more blocks

3:08:48 that you need to process and uh so um yeah that makes the the the whole uh trade-off a bit more difficult to uh to achieve. Uh so um yeah now I’ve now reached the point where I think it would be good to make a a demo. So uh I’m going to ask my co-founder Vanson to to join uh stage just so that we have a chat quickly and you can take a mic here and uh I’ll switch uh to a demo so that you you get really what 400 millconds really means in practice. Um so just about the setup

3:09:22 uh even though we have two microphones here it’s really my laptop that captures uh uh that captures the uh the conversation. So um >> so you you’re reing the tricks you see everything. >> Yeah. >> Hi everyone. Nice to meet you. >> Uh so how did my talk go? >> Quite good. I hope you share and people have been learning things about what we do. I’m always amazed how difficult it is. It sounds easy when you use it but uh >> yeah but we can even uh do overlapping speed. Absolutely. That’s what we usually do. >> 20% of the speech usually is

3:09:52 overlapping, which is kind of crazy to detect. >> Yeah, he keeps interpreting me in every meeting we have. It’s an ideal effect. >> For some reason, Ralph says something in the in in the backside because we found speaker 02 here. >> Please don’t back channel, please, Raul. [laughter] >> Okay. Uh I leave you there. Thanks. >> H So, so you you get the idea of what uh basically 400 millconds feels like. And uh yeah, so I’m I I’ve reached uh almost the end of my talk. I would like I hope you you you learned a few things in this uh in this talk. And uh at least the

3:10:25 three things that are I think interesting you’ll tell me about this talk is that uh I I try to convince you that uh batch and real time derization are actually two different machine learning problems. They have very different constraints. Even though the input and the output are the same uh solving it uh uh needs different approaches. Uh wanted to highlight as well that latency is just not one number. It really depends on many things. I mean you all all knew about that but in particular that for speakerization you see that the main latency is coming from

3:10:56 the algorithmic part because because of the way this speakerization task is uh is set up and uh yeah very proud of the work that uh the research and tech team did for to to handle this and uh yeah there are many ways of trading uh between latency accuracy and cost and uh yeah so uh thank you very much that’s the end of my talk And um yeah, happy to have a chat after if if you want. [applause]

3:11:27 Thank you. Thank you so much, Alve. Let’s give it up one more time for Alve, please. Nice. [applause] All right. Okay. So, this is a good moment because this is this was the last talk of this morning. Okay. Uh so, please uh you can go to the expo. We have a launch that is uh being sponsored by Sierra. So, shout out for to Sierra, please. [applause] Yeah, thank you for offering us some fuel, you know, to uh to to get going with this uh with with this event and to come back stronger in

3:11:58 the afternoon because we still have so many things to talk about. We’re going to be back here at 2 p.m. We’re going to have a fireside chat between Mistrol and Nvidia. You should not miss this at all. Uh so, I’ll see you here at 2 p.m. All right, enjoy your lunch.

4:50:11 Ladies and gentlemen, Please join me in welcoming to the stage your MC for the AI engineer Paris 2026, developer [music] relations engineer at Replet, Raul Chevrey. >> All right, and we’re back. Okay. Well, congratulations to to to us all, right? You guys are making it for the final stretch of today’s conference, day two of AI engineering Paris. Thank

4:50:43 you so much for coming back here. Uh we actually have a great great panel that we’re going to uh we’re going to start in a second. But how are you feeling? >> Yeah. No, no. Okay. You you know the drill by now. Come on. We’ve been doing it since yesterday. So, we got to bring up the energy. Okay. So uh we have two gentlemen here that um are going to come uh on stage. So please I would like you to uh join me in welcoming to the stage uh VP of compute at Mistral Yan Leger

4:51:15 and also uh senior manager solution architect at Nvidia Adolf Hall. So please let’s give it up for them. [applause] All right. And I kind of forgot to say that. Um, yeah, we’re going to be addressing like building the infrastructure behind the next generation of AI. So, we’re going to have like a little bit of a futuristic

4:51:46 view of uh what what’s needed and uh to to build the next generation of u of AI. So, thank you so much for being here, Adolf uh Yan. Um Jan, you were here last year at AI engineer. Do you remember? It feels like it’s been uh 10 years ago. >> Kind of a big year went by like our company was acquired in between. >> A lot has happened. >> Organizing like this conference together last year. Um and now we’re making it again.

4:52:16 >> Yeah, exactly. But it really feels like Yeah. One year in a in AI time. It’s uh feels like ages ago. >> I wish I would have been here as well. >> Yeah. you invited next year. [laughter] >> Yeah, that’s nice. So, Jan, when you were here, you were here on stage, you were talking about building for the future of agentic era. So, can you can you walk us through what do you think has changed in in the like in the interim? >> Yeah, I think like over the last two years I I’ve uh spoke about

4:52:49 infrastructure for AI. I mean, and over actually like the more more than this now. um last year was specifically a focus on sandboxes and how basically it’s creating a new category of infrastructure. Um and um we’re seeing it in action now. I think that’s what happened like the the one year we were in the beginning of this like motion of the agentic era and like uh the adoption kicking off uh and now um a

4:53:21 lot of the software uh is written by agents uh and we have a huge penetration of the agentic workflows overall. Um the but the GPUs are still like the the essence uh after all like I mean like we we have in addition to the GPU infrastructure we have CPUs which play into the mix now. Um that’s a key difference before between um the previous years and and now I think um so we’re still going in the same direction

4:53:52 like training large models it didn’t stop uh at ML was still training like state-of-the-art uh models on one side uh we’re doing large scale inferencing and the agent in KA is actually like kicking off even more the inference consumptions uh I think and which means the stack is even more complex. than before. >> So you’re saying yeah we were you were talking about send boxes and you’ve seen the rise of that type of instru infrastructure. So you’re seeing like

4:54:23 the progress in the load and I I think there’s no doubt in this in this new world. What do you think um is standing between us and the next generation of AI from a hardware perspective and from Nvidia’s

4:54:53 perspective >> when you are longer in this business and then you used to speak with compute with networking and storage and actually since this agentic workload is so dominant well where where where is it going to uh be executed and um I think often these days in this equation it’s forgotten where does the agenda workload go and that’s a CPU question um and um that is um that is I would say pretty

4:55:23 obvious if we speak in um model development so we have certain established architectures they have particular needs which drives basically the how the infrastructure and the the chips are designed for the for the next innovation cycle and uh they are certainly um it’s influenced by mixture of experts um and with its unique unique demands um of uh of communication. If we look into new modalities, I would say um we have I think the society and the

4:55:55 industry has made very good use of data. We I think we we we trained on every book in the world. We likely we soon train on every movie and every multimodel. But that’s not the end of the uh the game. I think there are other things uh um coming um which need to be tried out from uh um let’s say if if we go into this robotic domain um it needs to have feedback. So giving this feedback is a question of uh of simulation. So I think that would that

4:56:26 will pose new challenges. So we’re not there with it’s a very exciting uh journey. So I I just want to yeah double click on something you mentioned mixer of experts and it’s just a way for us to have very very large models but who can be efficiently uh being used at inference time right >> yes >> okay um all right okay um if gentlemen if you’d like to follow me I would like to talk a little bit about silicon hardware

4:56:56 um and um so yeah we know about Reuben the the latest generation of in leaderships and it’s now in full production, right? Um so yeah, some of the claims are are like that Reuben is 10 times um more efficient and it’s uh yeah, four times fewer GPUs also. So as uh as AI is evolving like we’re getting more efficient systems um yeah to train a model of experts. So how should teams

4:57:29 you know how should these engineers be building AI products thinking about the next generation of architecture given this cadence and this scale >> I would say try to follow as best because we on a the entire industry I would say is on a on a learning cycle so when I when I joined Nvidia there was an architecture called uh Pascal and Voltaar maybe you don’t know about that one but uh um if uh if we talk about data center economics and efficiency

4:57:59 what you also stress with uh more more tokens cheaper tokens etc. Then um we started a technological path uh in that generation and that is now uh seven or eight years ago and that continues. So um we need to work with that and the software follows these paradigms. So software and hardware goes hand in hand and uh in order to be of use you want to use it by means of the software and uh um so there are some dependencies when you say I want to uh make use of the

4:58:29 latest innovation for someone who um provides inference at scale obviously those are economic arguments and they need to be leveraged uh at best overall um we need to optimize um all layers of the of the stack. Um what we see recently is that uh um we used to think in in infrastructure for trading and inference and I would say that is that is going a bit more together also for an economic reason because uh um if you if

4:59:00 you make a pool it’s it’s static so in each pool you have some some waste >> and uh so that is that is I I think something which is uh uh go away and the the infrastructures for both domains um I expect um goes more together hand in hand. >> Interesting. Okay. >> Yeah, I do agree with with it all on like the the use the usage on on of the inference like there is actually two

4:59:30 kind of I mean in the past if you look at some of the talks I I did like two years ago I was already speaking about different kind of infrastructure for different purpose. um you leverage like uh training clusters don’t have the the same uh physical requirements. Uh they need the highest performance uh on the training side. Um and inference are workloads are a bit more tolerant. So they don’t necessarily need the latest generation of hardware. Um and we see this um in practice where for training

5:00:03 we might use the latest generation of hardware. For inference, we can use older generation of hardware. Um, which also means that like um when we design infrastructure uh on the physical side, we need to u keep in mind that like the usage on this kind of infrastructure is going to evolve over the time. uh you might like deploy a cluster uh typically and get the two first years used for one purpose and the three four year years

5:00:33 used for another purpose as Adolf mentioned like in terms of flexibility and ideally you can even go be more dynamic and and get your uh training workloads to be uh your training infrastructure to be reused dynamically for inference and so this is something which is a lot of software at the end in terms of orchestration to be able to reallocate dynamically the walkers. >> Okay, very interesting. So, we’ll need eventually two different types of infrastructure for um for inference and

5:01:06 for training down the line if I understand it correctly because the needs are different because we don’t need necessarily the latest type of hardware. So we I mean we actually were more converging maybe I mean we have specialized inference in infrastructure on one hand which uh if you build it like just for inference uh you can like tolerate like having not the latest generation of hardware it also depends on the kind of models you’re training like models now are as we know are old

5:01:36 of um different sizes and shapes. If you uh train a 1 trillion parameter model, it’s not the same as like an 8 billion billion parameter model, right? You don’t need the same training infrastructure in both cases. Um and actually we’re more I think like converging on the long term in a way. I mean on having a global pool that we reallocate uh between the kind of trainings uh and inferencing. >> Okay. Interesting. Um okay. So the next question is to for you Adolf. Um, I’m

5:02:08 going to talk about uh NVL72, but before I I dive into that, can you please explain what NVL72 is to the audience maybe who are not familiar with? >> In case you’re not familiar, um, AI is in the meantime, when let’s put it this way, when I started, my entry cost to AI was, I think, $150 with a GPU, I had the workbench, I could participate. Now, everything grew. Um, fascinating speed. So uh I would say that the smallest problem what Mistral has I think it’s it’s it’s not going to

5:02:41 fit on a single server anymore and uh um so at the same time uh the hardware pays attention to what is requested by the software and so uh multiple components need to work together in a most um economic and efficient way. Efficient means communication needs to be very very fast. And for that purpose um we designed NVL72 which is a a system in Iraq. And u um the the difference to a classic system is um that NVL72 is an

5:03:14 interconnect between them much closer than than anything of TCP IP. And this is uh necessary if you want to run the modern workloads. Um well and uh um I would say it’s the most economic way to address uh highly parallel uh trainings at this point. >> Okay. So yeah. So uh Nv72 is is a rack system and um so yeah, what does that mean for the rest of the stack? Does it mean that uh

5:03:44 yeah, we’re headed towards the rack being like the new server? >> Well, there’s a saying from Nvidia that the data center is the computer. So but uh thinking even bigger um uh but uh when you think about the workload you have to schedule various uh jobs on the units then um what is the most economic way of doing it and the NVL72 in itself is a unit of assignment and um so u economically there’s no need to say I’m

5:04:14 cut it in the in the in the middle you can do that um but you’re definitely leaving the sweet spot um of that uh of that system so in terms of I I would uh assign your your um your summary that it the computer lifts basically from a single system to a rack scale. >> Yes, it’s it’s going to be just uh like yeah just a bigger systems to serve our needs moving forward. Um, okay. So, yeah, I have a question for you, Yan.

5:04:46 Uh, since I mean like you’re you’re at the compute level, so you serve a lot of uh inference. And what do you think um that you will need moving forward from maybe from the chips or from a hardware perspective in order to maybe better serve your customers in let’s say in the near yeah in the near future? It’s uh I [laughter] think it’s well

5:05:17 it’s a it’s a tricky question like uh we I mean we have a way of consuming hardware which is like we take the latest generation of hardware for training frontier models. So which is like that’s uh what we need in general they they provide significant gain of efficiency and like to be at the frontier you actually still need it’s still like uh you still want more power. Um so in general we have this this mix and and we don’t only deploy NVL72

5:05:49 systems actually we do have a mix of both um system which is which with this high performance interconnect for training uh but we do also some deployments dedicated to inference and they might not be equipped uh with exactly the same system. Um so I think like we have um the the reality of the field is like our the diversity of infra infrastructure is increasing because um the AI wave has started like several

5:06:20 years ago. So we are starting to have like different generation and getting to actually a cycle of infrastructure which might actually resemble CPU in a way. Um the designs are completely different. uh but you do have a long-term like rotation in terms of hardware and you need to deal with this heterogeneous hardware. So um yeah I think in general like u um the chip manufacturers and Nvidia are designing what is like coming

5:06:50 next and uh we’re going to leverage this to to to train even more performant models. >> Okay. Yeah. Um okay I think uh I want to move on since we’re talking about inference I would like to talk a little bit about inference economics if uh if that’s okay with you guys. Okay. So, um, uh, yes. So, when Nvidia basically talks about, you know, AI factories, you always measure with tokens per second, per dollar, per watt, right? Like we

5:07:21 want the fastest way to serve inference. We want it at the cheapest cost and the lowest energy possible, right? Um, but yeah, you mentioned it’s different infrastructure for uh, yeah, for training and inference. But what do you think at what point we’re going to be able to justify the the the revenue of inference rather than just saying like um so we’re building basically for for training mostly. But when when are we going to uh

5:07:51 to be able to justify the the revenue for inference and say that yeah all that build out is not going to is going to materialize at some point. looking >> you’re looking at me. [laughter] >> Yes, I’m looking at you. Yes. Sorry, it was a question for if [snorts] uh nobody understood, but yes, it’s an Nvidia related question and >> well, everything what you what you what you use for for producing and making money that there needs to be some some it the economics will not change for

5:08:22 this one. So um it might have been uh a pattern in the past that you say I’m using training uh systems and uh when once the next generation is available I’m using that one for for inference for a less demanding workload. Um but still um the economics must be taken into account. Um and uh um now um I would say the inferencing workloads with the um it’s going through the roof. Um and um there is there is much more demand. I

5:08:54 myself I’m using it. I love to use it. I use it all the time. It it I don’t I don’t search in a classic way anymore likely many of you don’t do this. >> Um and I don’t want to miss it. Uh and so I’m leaving a trace um of this one and uh um so inferencing is uh um is also very important uh for us because we spend a lot of uh engineering cycles in making the software better all the time. Um so it has a um it is really at a top workload and so I would say um these

5:09:26 days um you need to take it into account um right from the first day. >> Okay. >> Yeah. And so if you go in this root like I think there was a risk which was before which was um a fear because the inference side didn’t materialize at scale and I think in the last year with the emergence of actually actually agentic workflows >> uh and workloads um we see inference taking off like I think like one one of the the thing which is discussed is like

5:09:57 how much like for instance if you look at software engineering how much do you spend on like AI tools to accelerate your software engineering efforts >> and so obviously there are discussion around token maxing or whatever like to to on this angle but um there is also just a reality that it’s a daily tool uh for anybody who is doing software engineering these days so um and we see the the revenue uh generation like uh from the the result of the previous

5:10:27 investments are paying off now >> yeah so basically the inference is is taking off and we’re going to see it’s just going to be uh increasing moving forward. So all this buildout is actually going to be used at some point by inference even though if it’s for for training at any time but uh yeah you spoke about token maxing uh and um yeah I reflect like we we don’t believe so much at uh at token maxing but we believe more um about um outcome maxing.

5:10:58 So we want to provide value for the customer and I think this is a good segue for my next question actually because yeah despite or when you quantify you know the the uh like we were talking about cost etc but at some point it’s not just a matter of cost it’s a matter of value to to the people who use it right so h how do you guys actually quantify that so I mean beyond just raw bruter commodities ities and you

5:11:28 know economics in this case. So how do we quantify intelligence? Uh >> so is a question for me or is it for >> this is a question for you? >> How do we intelligence? I think it’s a tough one like how do you quantify intelligence is more about yeah it’s about business outcome. So it’s uh and it’s highly viable. you can have like um with a low amount of token you can generate tremendous amount of of value if you apply AI right I think I mean

5:12:00 it’s one of the thing that we are doing at Mistro it’s not only about volume there are some activities which are um consuming a lot of tokens uh proportionally and there is a question there is all the efficiency and all the actually knowledge we still knew to to apply to optimize this processes because some of the workloads might be actually like using an abnormal amount of tokens uh for the outcome. Um and so I think

5:12:34 that’s where um all of the ecosystem can deliver a lot of value. If we look at like the the situation and like it’s not only about the M it’s not only about training it’s about like the complete harness and how do you actually like leverage the tokens properly uh to have a ratio of like token cost versus uh value. Then on the infrastructure layer, we’re just optimizing each layer. Uh but we are just providing like the tokens. Uh the value is really in the uh it’s

5:13:05 captured by the higher levels of the stack I think. >> Okay. Um well since I think since you are talking about you know uh the consumption of tokens and the ever rising the the consumption of tokens and harnesses etc. So I think like um it’s a good segue for me for my next question for Adolf about reasoning. So um yeah, one way to increase the number of tokens that we consume is reasoning, but it’s also a good way also to to have a better uh a better outcome, right? So which part you think of the stack that is

5:13:36 maybe the least prepared uh for all that uh you know tokens and that is generated by by reasoning agents. I would say I love it. It comes at a price. price is um we we consume more tokens. Um that means for everyone who provides infrastructure um I need the uh the memory to keep uh the the long contexts uh and I I need uh the

5:14:07 caching um to take the best out of uh reuse of these conversations and uh um so I could imagine that uh um we haven’t uh um deployed the latest uh or the out of possible everywhere. Um I see we are basically in in a phase where uh where caching um gets gets more important caching across um um in in a distributed systems um there are clear uh clear

5:14:38 signs that uh it goes into u um distributed serving um and this caching paradigm is important to this one and um well I think uh um we if we sit here next year I think uh we will we will review um the value of caching in that equation. >> Ah, super interesting. Okay. Um, I have one final question for the two of you. Okay. Uh, we said, yeah, we’re coming back next year. You’re invited according to to Yan, but we’ll probably be come

5:15:08 back coming back hopefully in five years from now. And, uh, um, so yeah. Do you think that we’re going to have any frontier models going that are going to be trained in uh, European AI factory? This is you. You can start with this one. uh Yan and uh >> yeah yeah I think I mean we’re acting on it at Mistro like we’re bringing up like 200 megawatt of capacity next year and 1 gawatt uh until 2030 so and um um and a

5:15:39 lot of it is in Europe so um and we are training on this infrastructure and actually like the current uh MR models are already trained on our infrastructure so I think in five years for sure we’ll have frontier models Uh, >> we’re announcing already. [laughter] >> Okay, great. Okay, so what’s the most overrated infrastructure trend in 2026? >> That’s my love. That’s for you. That’s for you as well.

5:16:10 Out of the blue, self-healing everything. What? Okay. And what’s the most underrated one? I think like uh in in infrastructure like it depends on which uh which uh layer we’re we’re looking at. It’s a a tricky question but like software

5:16:42 uh in a way. Um yeah I mean if you look at the purely infrastructure level uh infrastructure is software is probably like the the one of the underrated one uh software enabling infrastructure. >> All right that’s a good answer. I’m happy with that. So with this thank you so much gentlemen. Uh this is it for for for this chat and uh yeah thanks Adolf and thanks Yan. Please let’s give it up for Adolf and Yan. [applause] Oops.

5:17:13 All right. >> Well, the audience is part of the five years so mortal exercise, right? >> Absolutely. Yes. We’re all going to be coming back in five years. Thank you so much. >> All right. Let’s give it up for Yan and head off once more, please. [applause] Okay. And with that, we’re ready for our next speaker. Our next speaker comes from Moto. Please, ladies and gentlemen, please welcome to the stage Charles Fry. All right.

5:17:45 How’s it going? All right. Um, that was great intro music. I’m very pumped. I’m ready to do like a jazzer size or something. Um, and you need a lot of energy for low latency inference. Um, not just wattage, but

5:18:15 also uh grit. So, it’s an appropriate start. Um, so I’m Charles. I work on inference engineering at Modal. We’re a infrastructure platform. Um, and what I’m gonna tell you about today is our playbook that we’ve developed for optimizing uh low latency LLM inference. Um, so but before telling you how to make it good, uh, I want to talk about like what low latency inf low latency inference is and why it’s so important.

5:18:48 Um, so there’s this thing that got released recently called Jev. Um, that probably hasn’t been brought up every five minutes uh, today. Um, so, uh, Jev is a decision model, a a generic zeroot classifier that works really fast. Um, and people have gotten very excited about it. People have built these really cool demos. One of my favorite from Twitter is you like type a word and you get a pallet associated with it. Um, and the um, the person who created this demo, Matt Deloreier, said, “Oh, this is very cheap and fast. It feels like a

5:19:19 leap forward for creating new UX and UI paradigms.” Um, and the response, I think, for a lot of people who’ve been in this field in machine learning and natural language processing for a long time was actually kind of different. It was like, “Oh, I’m surprised that everyone’s so excited about this.” Um, and the sort of like basic answer here is that they made this intelligence that was always there cheaper and faster at the same time. and all of a sudden that unlocked new use cases if you aren’t

5:19:51 sweating that like you might run up to a usage limit or you might like spend $1,000 just trying to make an icon spin um and you won’t wait 30 minutes for that icon to start spinning then like all of a sudden there are more opportunities available to you um so um oh I forgot my corporate sponsorship statement yeah this is uh application uh Jeff and Typesafe building on top of our cloud platform So this is the kind of thing u that I’d like to help you figure

5:20:21 out how to build yourself. Um and so Jev sort of sits in like one of three basic categories for the kinds of things that people build with uh with language model inference with sequence inference. Um and my sort of like infrared take thinking on the back end here um is that you kind of want to divide these out into what is the kind of like figure of merit for their performance. The performance is so critical here. It’s what unlocks these UI UX u paradigms

5:20:52 that something like you know Jev for the original release of chat GPT was able to just like open up that was not there before. Um so this performance is super critical and for chat bots the sort of figure of merit is how many tokens per second can you get back to a user. So tokens per second per user um for a decision model or a zero. Um, and then the sort of final category, the sort of high latency inference category, these like data processors that might operate

5:21:23 on a whole bunch of data from your backend or might consume in Reduct’s case like you know thousands, millions of PDFs, they have a different sort of figure of merit, a different performance target, the number of millions of tokens that you can that you can process for a dollar or maybe billions of tokens. Um, so this is higher latency inference, so we won’t talk about it. Um we’re going to focus on the low latency inference. Um so in the sort of agent category um we’ve been working on serving the uh Kimmy models from Moonshot AI and the

5:21:56 like the difference between what you get from a sort of like offtheshelf let me take the description in SG Lang’s cookbook or VLM’s cookbook and serve it versus what you can get if you sit down and look at the workload and optimize um do a lot of configuration tuning do a little bit of uh um uh like performance engineering you can get um in our case we were able to get about you know five and a half times more uh efficiency tokens per minute per GPU but also three about three times faster per user. So um

5:22:29 about like 400 tokens per second per user instead of a little over uh like 130 or so. Um and this has allowed us to like serve this at large scale through a variety of platforms like open router and in partnership with our customers. Um and then on the low latency side, obviously you mentioned that Jev is running on modal, but also um we’ve uh written about um uh how we worked with Decagon to get the like lowest possible

5:22:59 latency for their like uh for their agent operating procedures that sort of do um this decision work um inside of their voice agent pipelines. Um so let’s talk about how that’s actually done. Um so the sort of like playbook we’ve developed. Um so the first step is that you need to understand the workload that you are serving and the hardware that it runs on. Uh and then once you have a clear picture what that is then you can start optimizing the performance. The

5:23:29 workload tells you what you have to do and the hardware is what sort of does it for you and it it says this thing will be easy. This thing will be hard. This thing will be 10 times as fast as this other thing. So in understanding the workload just to make sure we like I think a lot of people have been substantively consuming AI through APIs and not necessarily thinking about what’s going on inside. So just to make sure we’re all on the same page. Um we like you send in something like this thou shalt not create uh into a language model

5:24:02 inference server. Um this uh the uh internal representation of the model is calculated on this input in the prefill phase. Um that generates a few tokens and then you iterate that over and over again in the decode phase to generate the outputs and complete the commandment uh from the orange Catholic Bible that thou shalt not create a machine in the likeness of a human mind. And along the way, you store those internal representations in something called the KV cache so that you can uh retrieve it

5:24:32 later instead of having to recalculate it. Um, and that’s [clears throat] a pretty important piece especially if you want to serve low latency agent inference. If you’re serving something that looks a little bit more like Jev, this is less important because of the sort of structure of the workload, the structure of the requests that you’re going to receive. Uh so with an agent you see something like this where you have the the first user input comes in the LM responds and then later you get another request from the same user that has that

5:25:04 LM response and their first input in it and then you get another maybe a tool call from the LM and once that tool call comes back that also is going to have the whole prior context in it and so you have this iterative construction of the input sequence turn after turn. Um so some things about this um like this field and about the way LM and artificial intelligence work are kind of like flash in the pan. Um but this actually kind of feels very deep and seems like something that’ll stick

5:25:35 around for a while. Essentially the user, the model um and the external systems that it’s accessing are iteratively constructing this context and they’re creating something of increasing value over time. And so we should expect these kinds of interactive um like constructions to be part of how these sequence models end up working. Even once they become, you know, something inside of a robot moving around your house or uh once they start taking in videos and outputting videos, they’ll still probably look quite a bit

5:26:05 like this. Um yeah, now for a little something that’s a little bit more narrow in scope or momentary. Uh the good news is that right now um if you’re focused on high interactivity, low latency inference, you actually can usually run on no more than eight data center GPUs. So if you check out the really excellent benchmarking work from semi analysis and their inference X or inference max benchmarks, um this is maybe a little hard to see um because they’re very dense charts. This is like the least

5:26:36 dense version I could pull out. Um, you can see that if you want the highest throughput per chip, you want these big like megawatt scale, not quite a megawatt, these really huge [snorts] machines that have a bunch of GPUs in a in a tight domain with each other, an envy link domain. U, you need like what it says here, 40 of those GPUs, 64 of those GPUs. Um, but if you want to serve at the highest interactivity, if you want the highest tokens per second per user, you actually want to be all the way down here. Um, and if you look at

5:27:07 the points that are down here, you can see that they have four or eight GPUs. Um, so you don’t need um these fancy things, which is a huge operational win. Um, okay. So, we’ve understood like the workload that we’re thinking about and the hardware that we want to run on. Now, let’s think about how we’re going to optimize the performance. Um, so roughly the way that things work, um, is you want to target a bunch of big wins first. So speculative decoding is one of the biggest ones. Um you can get

5:27:37 a five time speed up or more. Uh quantization reducing the precision is another big speed up has quality implications. Um once you get those big wins out of the way, there’s like kind of a grind of mediumsiz tasks on the host side. Um like sort of regular CPU systems performance engineering stuff you got to do. Um that you wins 50% maybe as large as 2x. um the weirder the thing you’re doing is the more wins they’re probably going to be on the host side. Um and then only then do you spend

5:28:07 your time you know working on uh GPU kernels and the sort of thing that like everybody gets most excited about. Um so let’s go through that real quick at a high level like what do all these things look like and and when should you use them? So big wins first. Speculative decoding takes that decode process that we talked about where you generate um uh an output sentence like the LM is going to generate from modal is a to modal is a serverless computing platform. The thing you do instead of running this guy

5:28:38 one step at a time is you take something much cheaper to run and you run it um and generate the output tokens and you take a guess at what those output tokens might be. you speculate and the idea here is the same as speculative execution inside processors. If you have some like slack in a system um which is what happens when you run looms one token at a time uh then you can uh sort of use some of that slack to do work that might not be accepted. So run

5:29:08 instructions that might not actually be on the code path in speculative execution or process tokens that might not show up in the output like the word company here. Um, but you like it’s such a big win to be able to run these things in parallel, especially when you’re targeting high interactivity that this um is like very frequently a humongous win. Um, and it’s sort of linear in how many tokens you can get the speculator to guess. This is from one of our blog posts about speculators. Um, basically as you increase the number of tokens

5:29:39 that are accepted on average, your speed up goes up as well. And these are these are integral numbers like 4x faster, 8x faster. If you’re you know do performance work, you’re used to like you know sweating over 5% or 10% but these are like 500%. Um, and then one of my favorite things about this as somebody who worked on machine learning training from the beginning is that the difference between a point like this and as a speculator and model pair like this and one like this is that you um you customize to the

5:30:12 specific workload that you want the the the speculator uh or the target model to work on. you train the speculator to know how that model behaves on that data and all of a sudden you go from six times faster with it three times faster to five times faster. Um and this is you the bitter lesson of machine learning but apply to performance engineering which is exciting. Um people often ask like wait how can some dumber model guess what the smarter model is going to say? There’s a couple reasons why this works. Um the biggest one is that

5:30:44 there’s a lot of structure in outputs. Um, and so if you take something from your coding agent sessions like the uh the thing above and then you translate it into what the model sees, this becomes even more obvious. You have chat templating that has all these special tokens which are usually very predictable. And then this right here is a quote from the user which probably appears earlier in the transcript. So that’s easy to guess. There are only so many tool calls. um there’s only a few of them that are something you would say

5:31:14 you would do after saying you were going to read that file. And so this is like pretty easy to predict. Um but what the model the target model is doing is actually like processing these tokens and creating this rich internal representation that is going to be used for many tokens in the future. Um so you by being able to like guess what those specific values are with a good rate, you’re able to parallelize that um construction of their representation. Um which is pretty cool. Um, and that representation actually gets reused inside of current speculator models. They don’t just have to they’re not like

5:31:46 a marov model, an engram speculator or another language model from scratch. They take those rich internal representations from the target model and they uh they use that they use what’s what’s going on inside the model. So they’re able to kind of like uh bootstrap from that. So the specific architecture there was Dlash. um that’s the the sort of speculator architecture that we’ve found uh works best um and the one that we’ve invested the most in and gives us like large speed ups over both built-in speculative decoding in certain models and of course over the

5:32:16 baseline. Um after uh speculative decoding and getting that right, the next big win is quantization. So with quantization, you take the floatingoint representation of the weights or activations of the model and you decrease the bit width. So it goes from 16 bits to 8 bits to maybe even four bits. Um so this is a linear reduction in memory bandwidth demand and a linear increase in floatingoint operations per second because the the the bits are smaller or the the the

5:32:47 numbers are smaller. Um so this I say this requires full stack coordination because you have to go down to the hardware layer and be like okay am I running on a GPU that can actually make use of these things? Do I have a H100 or later for FP8? do I have a B200 or later for uh FP4? Um but it also changes model behavior. The magic of spec of decoding is that it changes nothing about the probability distribution of the model’s outputs. Um but quantization does and so you kind of have to go all the way back up to the application layer and ask is this still sufficient intelligence? Is

5:33:18 this are the capabilities still what I need if I quantize the model? Um, so with quantization, there’s this nice little visualizer on our LM engineers almanac that shows you like the effect of quantization. In this case, 4bit quantization on this uh image of a parrot here versus the original. I think maybe as a demo of when quantization does work. I think the people in the front row can probably see that this is a little bit um a little jank, but the people in the back row and maybe um you know people squinting at their phones

5:33:49 like maybe can’t see that that’s um that that’s been quantized. And so that’s the effect that you can rely on. Um sort of depends on what you’re doing. So those are big wins. Let’s talk about the host performance grind. So the key thing here is that the GPU does all the work, but the CPU can get in the way. the CPU is coordinating all the work that’s going on in the GPU. And you want to make sure that the CPU work that is going on does not block the GPU work. Like this is finished. Oh, now I need to do some work to decide what to do next. And then oh,

5:34:20 I’m going to start running GPU stuff again. That’s really bad. It’s a bubble or a stall. Um if you’ve worked with an event loop, then you’re familiar with this idea like don’t block the fast thing. Um there’s a key technique for this called CUDA graph capture that allows you to take like instead of launching like one GPU kernel at a time, you can just record all the kernels that you ran one time and then just launch them all at once. Um which is a pretty effective way to like cut a lot of this host work out. Um one tool for keeping

5:34:51 track of this is actually my my favorite is to just look at the power usage and the temperature of the GPUs. if you aren’t close to 100% power, you probably aren’t using the whole GPU. They’re very much power limited uh when they’re achieving peak performance. Uh and so in this case, we were able like just by eye noticing that this thing was way below its total GPU power usage even at this uh uh between this one which is at well over 2,000 watts and this one which is uh below it or kind of capped at it. Um

5:35:22 that doesn’t tell you what the problem is. You want to use something like a profiler to get in there. You want these traces. It’s a very empirical discipline. Um, and so the endsite systems or torch profiler, the tools that come from Nvidia for this are kind of your uh your go-to. And if you taking a look at that specific case, we’re able to determine that one of these GPUs is actually falling behind. All these guys here are all doing the same thing. The the highlighted blocks, they’re all running this deep gem. And these three

5:35:53 GPUs are all waiting on this one that’s slow. turned out to be like a NUMO region awareness thing. Um that was causing one of them to always be a little slower because it was on the wrong socket. Um there’s a lot of host level things that you can use PI Spy like the things are running in Python. So you could look at what’s going on in Python and figure things out pretty well. Um we have a blog post that talks about one case in particular where we took this like repeated sort of allocation and turned it into just a cache, a Python dictionary. Um, but I

5:36:24 guess you get to call it a CUDA IPC pool handle cache if it’s uh AIG GPU stuff. Um, but a Python dictionary that just held on to handles for these and suddenly that made the thing run 10% faster. Um, so this was mostly focused on like a one single replica like a couple of GPUs at a time. When you go to scale up, um, you’re going to run into routing problems. So you might predict performance before scaling that that just doesn’t uh happen once you scale up. Um, so one particular thing that

5:36:55 might happen is you might uh end up with like tail latencies that are much higher than you expect and that kind of show up seemingly at random. Um, and the answer is that they are kind of at random. They come from the way that you do your routing. Um, and this is especially important for agent workloads because of the KV cache. If you’re not routing carefully like with respect to the KV cache, then the difference between like a cache hit and a cache missed is maybe 30 seconds versus 500 milliseconds. Um, and so uh measured this in one of our

5:37:26 deployments and determined that there was variety of reasons causing instead of the you know four or five uh concurrent requests we were trying to aim for, we we saw this major overd dispersion of requests and the solution was just to make the router smarter. had to become much more stateful um and aware application aware of things like the load and the KV cache state but that was able to sort of pull in this uh overd dispersion and uh massively you know get rid of these uh spikes and massively improve performance.

5:37:56 Um so then and only then are you allowed to think about you know GPU performance and optimizing things on GPUs. In general kernels are relatively standard. They’re close to the speed of light. architectures are not changing that much. Um, and so there are really good kernels out there from Nvidia and from like treehouse group and others. So there’s only a few percentage points to gain there. Um, so that matters a lot at scale, but it’s not something to worry about at the like, you know, at the beginning of serving your own inference. Um, the specific tool for this endsite

5:38:28 compute allows you to like go all the way down to the assembler level and figure out where performance problems are coming from. Um, but before you pull out insight compute, I’d suggest you pull out um a marker and a whiteboard. Um, if you look at uh Tree Dow’s uh and together’s blog about their flash tension for kernel work. Um, the sort of the way they decided what to work on, the way they decided what problems mattered was just to write down the feeds and speeds like how much data is coming in, how many operations need to come uh happen on it and how quickly can we move it back out. And uh that is

5:39:01 something that you can do you know before um you know from first principles and this sort of applies across the stack. If you understand these things about the like hardware that you’re operating with then uh you can really um you can get a lot further um without confusing yourself and wasting money. Um great. So I said that we weren’t going to talk about high about high throughput but I actually lied. Um there’s a little piece of work that actually is just coming out today and that I wanted to share with folks here

5:39:32 at AI Engineer Paris. Um we’ve been thinking a lot about low latency inference at Modal because of how much people, you know, love coding agents, how much they love things like Jev. Um but there’s actually quite a bit of demand for high latency inference for things that look very different. Um, so the sort of prototypical example of this and that we’re going to talk about is AI SQL. So this is not the same as AI writing SQL, which my understanding is AI now writes all of the SQL in the

5:40:03 world. Um, certainly I wrote this um, but you know, this is this is not something I run in production. Um, but yeah, so this is not AI writing SQL. It’s actual it’s actually the other like direction, the opposite. It’s using SQL to write prompts for AI. Um, so the idea is, you know, you have all this information about your customers and about your products. Have you ever wanted to just ask Chad GBT like which of my customers might buy which of my products? Um, and that can be expressed

5:40:36 as like a relational join, the classic kind of thing that you would normally do in your database or your and in your transactional or analytic database. Um, but the problem here is that this guy here, this prompt is going to like bankrupt you if you’re running against like um a proprietary API. So like running this on your database might produce like 8 million queries all at once. Um, and then you know fire them all off and there goes your token budget. Um, yeah, that’s one way to achieve token maxing, but I think we now

5:41:07 all agree that that’s a bad idea. Um so uh the key thing that makes this a totally different optimization problem from the low latency inference is that you have this tremendous amount of structure. So this is a query plan for a little bit more complicated of a query that has a couple of joins in it. Um and this query plan says these are all the sort of documents I’m going to use to construct prompts. This is how I’m going to have them interact with each other. Um, and this is all the 8 million or 10 million or whatever prompts I’m gonna

5:41:37 I’m gonna generate and send to the inference engine. And the existing inference engines have no idea about SQL. Um, they like they only know sort of like request response maybe batch request kind of semantics. Um, so what if you took the sort of uh page out of the database book and you did like query planning and combine that with the inference engine. So we did that with um partnered with Treya Shankar at CMU uh to get about a 14 times speed up over the like baseline VLM on that specific

5:42:09 query that I just showed you and achieve like over a billion tokens per minute on a single H100 and finish this query at just like $2 to execute instead of like uh almost $30 with VLM and like uh quite a bit more with um something like uh even the smallest OpenAI models. Um, so this is actually open source. We just released it today. Um, this is the first announcement of it. Um, you can find it at the full stack data labs website and their GitHub. Um, and you

5:42:41 can read about it on our blog on uh, yeah, check it out here. Uh, all the information that I shared today is in greater detail on our blog and a bunch of resources about running your own inference on modal.com. Um, I’ll I’m gonna get pulled off the stage in about 30 seconds. So, I’ll just say the reason why I’m telling you all about this is both because I want people to run their own inference and because I think the contain the uh um serverless infrastructure platform that we’ve built at Bodal is the best place to run this stuff. So, when you go to try and run

5:43:12 your own inference, I think you’ll realize that modal is the right place to do it. um whether that’s uh by running code yourself or by taking advantage of modal auto endpoints um our automated system for producing custom inference deployments for uh for users. All right, thank you very much. [applause] Thank you so much, Charles. All right, let’s give it up for Charles once more. All right. [applause] Yeah, I totally didn’t don’t want to run

5:43:42 that query and bankrupt myself. That’s for sure. Um yeah, our next speaker is actually going to talk to us about the exact opposite. Uh he’s going to talk to us about how to run models in uh and in uh in smaller devices. And so he’s going to talk to us about edi. Our next speaker is Dominic Payak uh and uh he’s a senior director at ARM. So please join me in welcoming to the stage senior director at ARM, Dominic Payak.

5:44:13 All right. All right. Hi everyone. So yeah, my name is Dominic Pike. time at ARM and today we’re going to talk

5:44:44 about what happens when you run inference at the edge. So there’s there’s three things going on right now which I think are going to unlock some incredible opportunities. One is just the speed that small openweight capable models are becoming available. You combine that with the fact there’s tons of accessible hardware you can then put these things on and there’s this potential for devices that are a little bit more intelligent, a little bit more um aware of the intention of the user and deliver experiences that have not

5:45:14 been seen before in all kinds of different environments. And so yeah, I’m I’m from ARM. And so a little bit about ARM. ARM is the compute platform at the heart of uh devices all around you everywhere from highly efficient data center infrastructure. There’s physical AI with automotive and robotics, so hard real time safety critical applications all the way down to pretty much every smartphone you’ve ever owned, client computing and industrial IoT and emerging devices. And that’s the the stuff that me and my team have been

5:45:45 working on enabling new classes of devices to be created and come to market. And so I am a computer history geek. I love the fact by the way that this uh station F is on Parve Allen Turing. I don’t know if I pronounced PV correctly, but um it’s it’s fascinating to look at the history of computing and where it’s taking us today. So if we look back the past 80 years of how people have interfaced with machines starting in the very first days, you know, maybe

5:46:16 starting 80 or more years ago with punch cards and tape and then keyboards and then mice and pointers and touchcreens. Each generation has made computing easier for people to access and interact with. But irrespective every time it’s still the case that the user is having to give their attention to that device and have the mental friction of converting the thing they want to get done into steps that the machine understands in a form that it can be inputed into that device. And the

5:46:49 you know the amazing thing about the AI era is it inverts this relationship. Suddenly the machines understand us. Not I’m not saying in a conscious way but you know they can understand natural language and they can convert our intent into actions and tool calls to get things done. And so um in some ways you know we’re talking about edge inference in some ways this isn’t new. So there’s been object detection and CNN’s running on edge devices for for many years. So you know envision applications in

5:47:20 factory inspection or parking bay monitoring and all this kind of stuff. Um more recently embeddings LLMs and then agentic applications which I guess is LLMs that can reason strongly enough call tools and you know deal with context long enough to act autonomously on behalf of the users. These are just becoming possible today. I I think it’s good to step back a little bit and just look how fast this has happened. And so I, you know, I graphed this out and really this is kind of, you know, look

5:47:50 at um GPT3.5 and, you know, that was the first time I really experienced um what an LLM was and it was, you know, a mind-blowing experience. But if you look how quickly this capability is then being compressed into models with fewer and fewer parameters um up until the present day, you know, this can now fit in a billion parameters roundabout and then run on a smartphone or run on a Raspberry Pi in your hand. This is phenomenal. And I by the way, I know that benchmarks are not perfect and then an LU probably less so.

5:48:22 But it’s indicative of a trend which is going to continue. You know, if I’m here next year, this is going to have gone down even further. And it’s it’s not just general knowledge question answering, it’s reasoning too. Um and so here we’re looking at a frontier of you know what can you do with four billion active parameters. So the kind of compute on uh today’s edge devices and again you know this has gone uh from especially with mixture of experts coming into the picture gone from something which is now um exceeding what

5:48:54 uh Claude Sonnet 3.5 was doing when it was first released with its frontier GPQA diamond score. I think this was like a year and a half ago. Um this stuff is just moving phenomenally fast. And of course, it’s not just about capability. It’s about latency when we’re talking about human user interaction. Um, and there are also models now which can run comfortably on device which do uh speech to text in a latency which are useful for creating voice UI without any need to send this

5:49:25 inference to the cloud. And I think it’s very interesting to consider what drives you know these latency requirements. And it’s really, you know, it’s what people expect, what they’re comfortable with is the way we should be designing these things. I find this graph really fascinating. So this is an anthropological study of human conversation turntaking and the latency between speakers. And it’s interesting that it’s culturally dependent and language dependent. So, uh, Japanese is very quick and efficient in responding. Danish is very laid-back and very slow.

5:49:57 I don’t know if that’s because the verb is at the end of the sentence, by the way. Um, and then English is kind of in the middle. And so that gives you a kind of gauge of of where these things need to land. And so, yeah, so I hope you’re as excited as as I am about all of this inference being possible on edge devices. You may say, well, you know, we could do this on the data center already. Why, you know, why put it at the edge? And it’s it’s a good question. I think there’s some, you know, very very good reasons for this. Number one is efficiency. And so you know I think many people are already looking at their

5:50:28 token usage and thinking how how much of this could I offload onto a local device or a device on prem. Um there’s the fact that you can customize this. So maybe you want to use personal data or proprietary data to your organization and you would like to keep this uh inhouse. Um, and then there’s kind of this opportunity for attent a uh sorry, yeah, intentionaware devices where maybe they’re observing your speech or have cameras um and are constantly looking and listening. And a lot of people may

5:51:00 want to choose whether that stuff is shared off the device onto a data center or not. So I think these are some really compelling reasons why we would want to put the inference at the edge. And so I did say, you know, there’s something else going on here. And I think it’s this combination of you’ve got this accessible hardware. Um you’ve got these openw weight models putting this together I think is is a fascinating opportunity. And so I don’t know how many people are like hardware makers or into hardware here. Are there hardware people here? Can I have a show of hands?

5:51:32 Good. Okay. I’m glad to see that. And so, you know, it’s been notable that for many years now, people have been building pretty amazing um hardware innovations using these composable modules from like Raspberry Pi or Arduino maybe in the first instance and then uh creating totally new designs based on this stuff. And I think that combination of hardware innovation plus these open models is going to open up some really interesting potential. Um, I think the observant among you may notice

5:52:02 that the micro duck is in the corner there. And you know, my my favorite example of this combination because I don’t think it’s really uh as widely um, you know, it’s not happening yet as widely as it could do, but a really nice early example of this is the hugging face robotics. Oh, sorry, hugging face pollen robotics, Reichi Mini. And so I know Reichi Mini is from France. Um, I was very happy to be on the beta uh, tester program for Reichi Mini. So last Christmas with my son, I built Reachi Mini and I was really happy to see. So

5:52:33 inside of this thing, it has a Stewart platform so it can express emotions with its head and it’s got a bunch of servos that um move the head platform. So this has ARM Cortex M0 in there. They have actuators on the antenna. They have a Raspberry Pi compute module 4 for the Wi-Fi version. And so with that you can either run some applications on the device itself or you can use that to then call inference services off device. Um and so that’s super cool. But it’s

5:53:03 not just hardware. It’s also an SDK which is open source and it’s a community of developers creating really cool applications for this platform. And here’s one application example I would like to highlight. So this is um by our very own Marco Domingo at ARM and he created an app which basically it sits on your desk and if you use your phone or like start doom scrolling or getting distracted it tells you not to. And this this won a prize. So this won him a VIP

5:53:33 ticket to Nvidia GTC in uh San Jose this year and a DGX Spark. Um and it was my Reichi Mini. I have to say I’m not bitter about that but well done Marco. It’s a cool app and actually if you go on the app store it is there today on the it’s a hugging face space where they keep reaching many apps. So in the design of these types of ondevice applications you know there’s a engineering challenge because you have constraints right so you have to think about the available compute and memory and oftent times you’ll have to divide

5:54:04 up this task to make best a use of the available hardware. So for a vision application, it’s quite common to maybe have like a motion gate. Uh maybe this triggers object detection and if you classify things you’re interested in, you grab a region of interest, maybe you do image uh embeddings and maybe only then do you go and then fire off a VLM. And so these tasks may be carried out by different types of processor on device and in some cases may then go off to um other devices entirely or even to the data center to be run. Um and here’s

5:54:37 another example. So this is the app I made during the beta test of reachi many and this is running entirely on a Raspberry Pi 5. What I love about this is you you know you say something and reach will immediately turn his head and there’s something magical about embodied AI in the way that you know a smart speaker may just bong or the LED lights up but having a character turn its head to face you and then respond is a really cool experience. And um I think in doing this, you know, I was using a Raspberry Pi which is definitely capable of running a bunch of different LLMs, but

5:55:08 just because of the latency constraint, I chose to use embeddings and then do a semantic match and call tools against this. So you could still ask it the weather or the time and it could generate using text to speech on device also and reply. And here’s Marco’s demo. And by the way, I forgot to say, so in the main expo outside after this talk, Marco will be showing this um demo live and you can ask him questions about it. Fundamentally, what we have here is he has his DJX Spark, so he’s got a more capable device uh from his prize winning

5:55:41 um and he’s running um another interactive agent like demo. Um this time he’s using Mistral Minstrol 3 on device and he’s doing speech to text. Um, and then he’s uh creating responses, synthesizing a response using text to speech and playing it back. In addition to that, he’s doing a bunch of tool calling. And so he has this ability to, you know, you can ask it what it’s saying. It’ll tell you, you know, actually, maybe I’ll play the video.

5:56:12 That could be a better way of doing this. Let me just do that really quickly. >> Okay. >> Yes. >> Can you greet the room? Hi everyone, I’m Reachi. Nice to meet you all. >> I’ve got a badge here for AI engineer and let’s see if Reachi can do anything with it. Hey Ree. >> Yes. >> Can you meet the attendee? >> Show me your badge. Yes. Got it. Give me a second.

5:56:43 Hi Dominic. Good to see you at the AI engineer conference. I see you’re from ARM. They’re known for designing risk based CPU architectures powering everything from smartphones to supercomputers. >> Now finally, can you dance? >> How was that? Too much? Want me to dance again? See you later, everyone. Reachi’s off to recharge. Have a great day. >> All right. >> So, I love the character that um and

5:57:15 personality that Reachi has through minstrel by the way. too much is a really cool uh demo. If you go outside, like I said, you can actually go and see if it can read your badge and tell your fortune, etc. It’s really cool to actually interact with it in person. So, those are a couple of examples, but I did talk at the beginning about intent. And so, you probably heard about the intention economy, and you know, the attention economy is, you know, you get people looking at apps and there’s adverts and and stuff like that. The intention economy is understanding

5:57:47 um what people want and providing solutions to it without them having to spell out everything. I think that’s the way I would describe it. And here’s a little example of an interaction that could be possible through this intent. So you know if you have for example reaches a bed light bed night lamp there um that’s also connected to your home assistant system for example um if you ask it you know you tell it you’re done for the day it may not be able to interpret that right if it understands your preferences in terms of uh you know

5:58:18 AC temperature or lighting it can go and act on it if it’s got ambient context and if reachi is maybe looking or listening and remembering has memory um about the context it’s it’s receiving. It can maybe see, oh, your partner’s asleep. And so maybe Reichi doesn’t say anything. It just nods and it kind of dims the lights and it goes to sleep itself. And I think, you know, this ability to deliver a better UX with fewer interactions with a device is really the key to this thing. This is just one small example and I think there

5:58:49 are thousands of others. And inherent in this is the fact that you know there there is an agent running here. So we talked about vision and audio and sensor streams and maybe integration with with home assistant. These are effectively events or maybe state that can be uh queried from an agentic system. And of course, you know, with with the agent, there’s a harness which is running on a CPU and that CPU is doing a bunch of work managing that context and and the memory. And then I

5:59:19 think the uh key thing here is how much of the inference can be done off or off of the device. And I think this is like a an orchestration problem which a lot of the industry are focusing on today. Certainly, you know, task aware routting and being able to judge whether a task is possible to run on a device is is a really key one. We’ll look at that in a second. And finally, you know, we’ve been talking a lot about consumerbased applications and how people interact with devices. There are a ton of

5:59:50 embedded systems out there that have computers that actually, you know, maybe they’re in industry or smart green houses or, you know, all kinds of commercial building systems where um they’re mainly monitoring sensors or maybe doing actuation but with limited uh interaction from a person. And so I really like to think about this idea of of those thousands of embedded systems that are out there. Is there a potential for, you know, an agentic embedded system? So I’m not saying that the agent

6:00:20 carries out all of these tasks more that the agent takes the role of a supervisor and when things happen out of band in otherwise in other words you know if there’s a fault that the designer of this system hadn’t anticipated can these agents improvise to go and fix them and the kind of thought experiment uh I think is really interesting is you know I I’m also a space geek right so Voyager one is like I think it’s the furthest man-made object from the earth right now and it’s left our solar system and it’s

6:00:50 um you know there was a fault a couple of years ago with one of the memory chips on this device and NASA were able to create a patch to re you know reroute this basically and and fix the error but they had a roundtrip delay of around 22 and a half hours and it was like something like a in the region of 100 bits per second communication to do this and so you know it’s interesting to consider could agents be used to preserve these systems systems and monitor them um and keep them in good

6:01:21 health and I think this is one of the really interesting avenues for exploration on the tip on the topic of hybrid AI um so you know there are a ton of interesting uh research papers in this area and there’s some open source projects you know um light LLM or root LLM um that you can apply to this challenge of you know can inference run on this constrained device in a model that fits there do I need to send it externally internally to a more capable piece of hardware. And so I was running some experiments the other day that you

6:01:52 know there’s a couple of different strategies. I think that the embedding kernel is the the quickest because you don’t actually have to run the prompt. You’re really just uh you know you have some kind of calibration set. You look at prompts that successfully ran locally and ones that didn’t and you see which one is more similar to your you know prompting question. There’s other solutions out there like calibrated uncertainty or even decoder state probing which are possible once you’ve actually run inference. So you’ve already drafted uh a response to the prompt and it maybe then you need to

6:02:23 kind of defer this off device to be run um if the confidence is low or if it classifies as something which would would have failed. This is super interesting stuff I have to say. You know these evals are really just looking at single um single prompt uh eval sets. there are, you know, there’s pinch bench uh chlor is it chloral? There’s a bunch of evals looking at multi-step tasks. Um I tend to think, you know, it’s really your own application which is the best thing to consider in these cases.

6:02:55 So, I’ve talked a lot about what we can do with this combination of really efficient models and accessible hardware and just like the inspiration of creating devices that are a little bit more I think you know a more more uh considered and less obtrusive. I think this is something that would benefit everyone. Um but of course you know I should also talk about the hardware beneath a lot of this stuff. And I mentioned at the beginning I work for ARM and ARM has uh processor solutions that span from these really low power um

6:03:27 wearable or you know sensor node type devices with our CortexM and ethos up to Linux-based platforms with Cortex A which you might find in um in Jetson or in Raspberry Pi right the way up to data center infrastructure and our focus is providing a common software framework and uh we’ve recently announced an AI portal which has optimized models and also performance optimization tools. And one thing, you know, I would definitely encourage you um you know, I’ll show

6:03:58 this QR code at the end. Uh please register uh become an early access customer to this thing. We’re interested to get your feedback and improve this for the applications you’d like to develop. And uh just a couple of examples. So you know, we’ve upstreamed uh kernels for Clyde AI into Llama CPP into Onyx runtime. So when Gemma 4 came out, it ran optimally on SME2. Um, but it’s not only the new stuff. There’s also consideration for uh really widely used kind of established models and

6:04:29 hardware. So I really like this example which we have just released or due to release very shortly um which is Ultralytics YOLO 26 kind of optimized specifically for the Raspberry Pi 5 um ISA. And so they got 1.87 eight, seven times faster inference. I think the key thing to point out here is, you know, typically when you look at the FPS numbers for some of these models, they’re occupying, if they’re running on CPU, all four cores. And really in an

6:04:59 application that is not ideal. You would like some headroom to do other stuff. And this kind of optimization is the difference between, you know, having two cores free to run your application, run other kinds of algorithm on that output. So this is really interesting stuff, I think. Okay. So, if you would like to get more resources, there’s that QR code again for developer.arm.comai. Please sign up. Uh, yeah, there’s early access to our optimization tools which I would encourage you to get into. Um,

6:05:30 also I’m really happy to connect with you either on LinkedIn or I’ll be out by reachy later on. And so, yeah, just to say again, so Marco Domingo, I think he’ll be there at 400 pm. So, not directly after this, but at the break. uh he’ll be on the the expo stage with Richi Mini if you’d like to go and chat with him and interact with Richi. All right. Thank you very much. Cheers. [applause] I feel connected to co

6:06:48 How you feeling everybody? Yeah. Good. Yeah. Okay. You know what I’m going to do, right? How you feeling guys? Okay. I think the next one is going to be great. Yeah. Well, thank you so much uh uh thank you so much do Dominic for the presentation. Now we’re gonna move on to our next speaker who’s going to talk to us about agentic map produce for large scale coding agents. I’m I’m super curious about this one. I want to see what what this means. But

6:07:19 please join me in welcoming to the stage VP of engineering at Cognition Yanis Sorakis. [music] Okay, let’s see. Oops. Am I connecting to the

6:07:52 right projector? Classic. Please give me a few seconds to sort it out. Uhhuh. That’s good. That’s good. It’s promising. And I need to see how I extend my screen. Ah, slideshow. Excellent.

6:08:23 Thank you very much for having me here. It’s such a great pleasure to be speaking to AI engineering in Paris, this great audience. Um, very excited about what I have to show you today. Um, an approach on tackling very large codebasewide problems. And uh, yeah, let’s get going. Um, a few words of intro for us. We are Cognition. We’re an apply AI research lab. Our founding DNA is in um AI research,

6:08:56 competitive programming, competitive maths. We’re based in San Francisco, but um we are now all around the world um including London. This is our European base of operations where I’m based and we are the makers of Devon. um few so Devon Devon is the first um AI coding agent in the cloud um which now has grown as a platform both in terms of kind of like different form factors um both local and desktop and also in depth

6:09:29 in terms of the different integrations and agentic personas across the SDLC that we provide. Um we deploy Devon across the most like sophisticated um organizations in the world. They range from um systemically critical banks, Goldman City, Santandere all the way to the US Army, big tech um smaller like a fastmoving technative startups

6:10:00 and uh we love tackling like a really really like difficult uh coding tasks and I want to reflect on our experiences on sort of like the classes of problems that one uh can See let us consider let’s start by think two categories okay let’s call them kind of like localized um agentic workloads which are basically kind of like take your median session with an agent I want to implement a a

6:10:30 feature or given a particular area of the codebase I want to kind of like write some tests or augment the existing test coverage um given a particular like slice of my DB proxy I want to optimize queries and so on and so forth this concerns work that’s kind like a like a bounded problem it is like a set of relevant files in the codebase okay that basically modularize the whole logic and

6:11:01 equally importantly if I were to tell an agent what what to do I will either point them explicitly to the files or if I don’t do they can trivial go and find the relevant area neighborhood of the codebase to carry out the task. Now let us consider what I would kind of like simply call codebase wide task. And by the way codebase wide doesn’t only mean like a single repo. You can have

6:11:32 multiple repos. Let’s take for example a case that we’re going to revisit again again today like security scanning. what is the security profile of my application? We know that there are like the individual security kind of like a quality levels of different modules of the codebase. But what is equally important is the different kind of like the ways are interplay and how those things can evolve in the real world, how they can connect in order to reveal

6:12:03 vulnerabilities. similar in a similar vein extracting common logic. Um we work with clients that have like really really large code bases. They want to commonize ways they treat arithmetic um financial formulas or even kind of like infrastructure ways of interacting with the database. Um they’re all sort of like first of all it is a search problem but it’s also a problem of sort of like understanding and finding all the different nuances

6:12:35 okay by which you need to a common library needs to be extracted need to conform to and so on and so forth um there’s a complex interplay between the components take the security example the security quality of my individual let’s say microservices is okay could be kind of like a you know okay I I have like if I were to look at them in isolation low um priority vulnerabilities they can be in the backlog but when chained

6:13:05 together they can give rise to a critical exploit and then therefore like for those tasks the result is trustworthy when you consider the totality of the codebase or the totality of the deployed application or sets of applications. All right. And of course, one can say kind of like, yeah, that’s why we have end to end tests. Okay. That’s why kind of like in our SDLC, we have the different quality gates and so on and so forth. Absolutely. Our goal is always to to

6:13:39 shift left, right, to to catch those things and be aware of those things as um as soon as possible. Now if if I were to double click a little bit on sort of like uh what the challenges an agent could face in those codebase wide tasks I’m telling can you please find I want to find instances of a particular formula of a particular logic that appears in different variations across my codebase because I want to extract to a common library. Okay, there is a

6:14:10 challenge of finding the work in the first place. Okay, it is a search problem. Okay, agents will spend a lot of time searching and backtracking like burning tokens. Okay, number two, polluting the context. There is a great greedy amount that’s going to build up that is going to be competing for attention. Very very variable and diverse type of context. And number three, what is effectively our stopping criteria? Okay. Again, to make this as strong argument

6:14:41 as possible, one will say, “Yeah, of course, we’re not going to solve those big problems, okay, in a kind of like in a single session. There will be some notion of agent orchestration. Okay, there will be some notion of okay, we’re going to have kind of like an the agent outer loop. We’re going to break down the problem into different tasks. Okay, we’re going to define a dag how those things need to be completed and

6:15:12 then we’re going to have some notion of inner loop and the agent. We have to build all that stuff. Okay, because obviously no nobody is going to oneshot prompt hey kind of like please fix security vulnerabilities. Absolutely. My argument is to today is to present to you a framework okay how we think about the problem that gives the best um both in terms of outcome performance and cost and we make this okay as a first class notion in our product

6:15:44 um I touched upon this before kind of like single agent reasoning I’m not going to labor too much on this point okay both in terms of kind of like time spent searching context pollution and so on and so forth single agent problem does not solve very very large codebasewide tasks. How we approach this? Okay. So um we propose okay a variation or an augmentation of the map reduce framework an idea from um distributed systems. How

6:16:17 many people here have worked with map reduce? Awesome. So super super quick example map reduce introduced by Google was a way to solve to perform analysis on like a really really large data sets that they are distributed that they live on a distributed file system. Okay. So you if the classic example that we give hey we have a huge textual corpus on the distributed file system um different

6:16:48 files and we need to do some lexographic analysis I’ll keep it simple count the number of words on each book okay will be a series of map functions these are simple stateless as in pure functions that they kind of like we count each book they can run in parallel and there’s a reducer step that consolidates the results. In a very similar vein with a adaptation, we want to port this into

6:17:20 the agentic coding world. Okay, where instead of like huge data sets, consider huge code bases. Okay, an overview of how this thing would work. Okay, firstly, there are two steps that we introduce at the beginning. There is a main agent that creates a plan. The plan is takes your prompt or takes including taking skills, taking existing knowledge from the repo. Okay, whatever you provide and performs an anal and those

6:17:52 all those considerations in order to create a series of selectors. Selectors are areas, okay, are deterministic functions, okay, when when you run them, okay, it will create a catalog of your codebase with candidate areas for investigation. Number three, for each of those candidate areas, we spin a series of parallel sub aents where they look these areas and only these areas and they

6:18:23 perform and then the analysis. This is the map step and then there is the reduce step. The main agent again considers the results the combined results the distilled results from all the different sub aents. Okay. And performs some reconciliation some consolidation analysis. This this can be duplication prioritization or some more richer analysis on the combined result. Let’s kind of like double click a bit on

6:18:54 those concepts. Okay. And I’m going to use I’m going to use security. Okay. As kind of like a motivating example. Okay. I have this large code base. And what is the security kind of like the security quality security level of this codebase. Okay. I will provide the threat model. I will provide any additional context on the codebase. Okay. What the pre-work step is going to do is okay, we’re going to write this

6:19:25 determin a deterministic search function that will identify areas of the codebase of interest in the context of security. You folks can imagine API boundaries um where tokens are being minted and be passed around um SQL entry uh code entry points and so on and so forth. The outcome of that is to produce basically those deterministic selectors. They’re going to run and they’re going to produce like a series of batches, okay, or a series of shards of the

6:19:56 codebase where we can then run each one of those sub agents. Okay, those habages then they have a very nice clean and bounded context which they can perform the investigations and reports and so on and so forth and produce kind of like a bounded outcome for that particular area of the codebase and then the results go back to the main agent where we do kind of like a holistic reasoning across all the different findings

6:20:26 in the again in the context of security to make it like a little bit more concrete. probably we’re going to duplicate number one. Number two, there is a prioritization okay of which vulnerabilities are the most critical and there is a ranking of them. Number three, but this is crucial again in the context of codebasewide tasks. Um presence of chained vulnerabilities. Remember individual agents may tell you that their individual narrow part of the

6:20:56 codebase has some low um um criticality finding. But what the reducer step can do is can assess combinations of those low vulnerabil low criticality findings. Can they produce something much more serious? And again with the right hardness, okay, um Devon can run, okay, can actually test, okay, and try to simulate those attack paths. And to do it kind of like a little bit more visually, we treat the

6:21:27 tend to treat of code bases as basically a graph. Okay, all there’s all those references between um the different parts of the codebase. selectors look for off areas, look for API entry points, uh data SQL entry points and then from those selectors that they have run cheaply and deterministic in the codebase, we deploy okay individual devons, individual agents that perform security analysis for this particular narrow bounded path of the codebase

6:21:58 and then on the reduce step Devon will produce something like this. Okay, where it will sort of like number one triage, number two the duplicate, number three run and test and try to recreate those vulnerabilities including kind of like false positives as well. Um so yeah mention about reproducing because obviously at the end of the day you don’t want to ship kind of like tickets because we know whenever we have a release you know we run second

6:22:30 vulnerability from tools there are like tons and tons of things to turn through we want to be shipping kind of like legit PRs against a known vulnerability so recreating is extremely extremely important now we’re pushing the PRs and just want to comment Okay, a little bit on kind of like the relationship between cost and performance. Okay, of course kind of like you can run very kind of like expensive agents. You can run them against really large code bases. Okay,

6:23:03 it’s what is extremely extremely important as I mentioned earlier is to think hard and have the best possible ways okay to mitigate the cost of searching. Searching is a huge problem in large code bases and context pollution and context overflow. This is solved by the Devon and the main agent in the agentic map reduce writing those deterministic selectors and creating lots and lots of banded context which end up being much cheaper and end up having a better

6:23:35 recall profile rather uh compared to competing models. Okay. And we’ve run this on 50 data sets, 50 vulnerabilities from kind of like a GitHub and they run across like a multitude of languages, Golang, Python, Java and so on and so forth. Um, another thing that we should think about, I talked a lot about security agentic map reduce in the context of vulnerability detection and remediation. Um, think about it about it can be a

6:24:06 generalized concept across all different codebasewide workloads. Okay, dead code removal I mention a lot extracting common common logic into some centralized library from a very large code base. Um, dead code detection and so on and so forth. Okay. And this is something that again we from our research we have kind of like moved it to security and we elevating it into a first class feature um in our product. Um we are as I mentioned kind of like

6:24:38 earlier we are um applied AI research lab we doing work on areas like aentic map reduce building our own coding models 2.0 know research on model harnesses like dev infusion um agentic infrastructure um please check out our research blog and if you have any questions um this is my email I’m looking forward to uh your comments and see you around in the conference thank

6:25:11 you very much [applause] thank you so much let’s give it up for Janice once more please [applause] all All right. Okay. So, we all deserve a break right now. I know that it’s been a long day for everybody, but we have a great line of of speakers still. Okay. So, we’re going to be back here at 4:30, please. And uh enjoy your break. All right. See you in a bit.

7:14:50 Heat. Hey, Heat. Heat. Heat.

7:15:38 Ladies and gentlemen, please join me in welcoming to the stage your MC for the AI engineer Paris 2026 developer relations [music] engineer at Replet. Rahul Chevrey. Okay. All good. How’s it going, guys? Hey, this is the last stretch. Okay. So, this is where we need to gather all the

7:16:09 energy and we get going for the last stretch of this AI engineer uh at Paris. And uh I’m so excited about our next speaker. So our next speaker actually you don’t need me. He doesn’t need any introduction. He was known as the Typescript wizard and then uh you know he was also known for his work at Verscell before he started talking about skills and then it turns out that he has really strong opinions and good opinions about AI. So our next speaker is going

7:16:39 to talk to you about how to fix your PI bottlenecks. And my god, I relate to that so much. Anybody relates to that? Seriously, like who’s who’s having like so many PRs coming their way and then they they can’t review any of them. So, um, well, without further ado, please join me in welcoming to the stage Matt PCO. [applause] All right. Hello, folks. Having a good conference

7:17:10 so far? having a good conference so far. Okay, good. So, I’m here to talk about fixing the PR bottleneck. And this is kind of a grand title for trying to fix the thing that most organizations struggle with, I think, and have kind of historically struggled with before AI. You know, we have always had huge numbers of PRs just laying around that no one’s bothered to review. And this has now increased massively because of the new strains on us because of AI imposing all these

7:17:41 weird constraints. And to do this, I’m going to use the rubric of my skills, which is you’ve kind of heard about, maybe you’ve used them. And I have a couple of new skills to announce that are going to hopefully improve the way that you do PRs, improve the way that or improve the speed at which you can review them and do them. Speed. Now we’re being pushed to do more with less essentially or more PRs, more work, more stuff. And this is kind of the central

7:18:11 promise of AI that we’re going to be able to use these agents to scale ourselves up to do more work. And this has resulted in the software factory, probably the biggest buzzword of the day. Everyone’s talking about software factory that I chat to. And I think of a software factory as primarily something where instead of the human initiating all of this work, we’re going to pass some of that initiation, some of the initiation is going to be done by agents. And I think of their

7:19:04 maybe from there you have a classifier like Jev come in and Okay. Okay. Let’s turn that into a fix or turn that into a reproduction or maybe I ping people straight away. And maybe you have other things. Maybe you have planet scale hooked up. So it gives you query reports on the slow queries on your database. Maybe that then triggers a different um thing of your software factory. All of this is not humans triggering it. It’s uh deterministic code triggering it. Right? And so these accelerate your software factory. They push more code through it. But then you need breaks, right? If you just have permanent

7:19:37 acceleration pushing stuff through your factory, you’re going to end up with a slop cannon, right? You’re just going to end up with a ton of slop crappy PRs that you’re not going to be able to touch or review or even freaking look at. So, you need breaks. These are mechanisms that slow down, that increase quality, that make sure that your codebase doesn’t turn into a software entropy nightmare because code is the environment your agent operates in. And if you have bad code in your codebase, that is going to beget more bad code.

7:20:07 And so I’m going to talk about these three breaks in this talk and talk about how we can use them to counterintuitively go faster. So automated checks, these are the deterministic checks in your repo that we’ve had for thousands of years or you know since the 50s. Deterministic checks where we have linting and tests and type checking and code quality metrics. All of these things going together and they all work the same every time. Layered on top of that we have automated review. So

7:20:37 we have agents who look at our code and say okay you know these are for the things that the tests didn’t catch or this is looking at the structure of the codebase in general. And then on top of that the third layer the final layer is human review. So people looking at the PR. And these three layers form this kind of cake that we end up with when we get to human review. And so more speed of course means more PRs. And so the goal here is to make human review faster by leaning on those first two phases.

7:21:09 And we got to stop the slop. That’s the first principle here, which is if you raise the quality of the code that you’re shipping, you’re going to end up doing less human review because it’s just going to be better work. And so you’re going to end up needing to u make fewer interventions. So automated checks. Now automated checks are cheap. That’s the cool thing about them is that they don’t cost tokens like automated review does. They don’t cost human effort. They just cost CPU cycles. So these are for instance, you know, you run your tests on every code change. Maybe those tests do incur

7:21:41 some tokens because let’s say an agent uh creates a bug and the tests catch it, then you need to spend some tokens to go and fix it, but those are tokens pretty well spent in my opinion. So checks are cheap. That means you can layer on loads and loads and loads of them on your repos. you’re probably not using enough of them or not being creative enough with your use. But checks can lie. Does a green CI mean that the code is ready for merge? No, it does not. And so we’ve always needed on top of these

7:22:12 checks some extra layer to figure out if there’s anything catastrophically wrong with the code before we ship it. And so all of the other phases, the human review and automated review, these are lie detectors. These are for finding lies in the automated checks. Now, I want to show you some of these lies first of all because this helps when we’re thinking about code and thinking about automated checks to see how bad it is and how bad things can get. The first is tortological tests. A test that just reasserts the implementation. Opus 5 got

7:22:44 addicted to these. I don’t quite understand why it would have for instance x post character limit equals 280. Can anyone guess the test that was written to test this behavior? Right? You’ve probably seen this a thousand times. This is real code from agents or from stuff that I found my agents doing. It said expect x post character limit to be 280. So the implementation looked like that and the test essentially reasserted the implementation. That is a tortological test. And tological tests are bad

7:23:16 because they’re extremely structure sensitive. They’re very sensitive to the actual internal workings of the system. So it means I cannot change that constant without a test failing. But I cannot rename that constant without a test failing. Like I have to do is so tied into the structure of my system. And I found another one which is even more egregious I would say. This is incredible uh test. What it’s doing here is it’s essentially testing whether two

7:23:47 things in the UI appear in the right order. So it’s checking that the pitch detail page uh or rather the video section comes after the content plan. What it does is it doesn’t render it to a screen. It just reads the actual file the module into its own memory and then it’s finds the right thing. So, finds content plan, finds videos, and then it expects the videos to be after it in the source material, which is crazy if you think about it because I can just like change the way the source material looks

7:24:18 and this test will fail. It’s too sensitive to the structure of my codebase. So, that’s another way that automated checks can fail. And there are also or sorry, automated checks can lie. There are also tests that literally cannot fail. And this will feel familiar to you if you’ve abused mocking in the past or abused various things. For instance, here we have a use audio boost function. And this use audio boost function uh its internals use the audio context API in the DOM. Don’t worry if

7:24:48 you don’t know any of this, but what we’re doing here is we’re just stubbing it out with some fake methods. And it turns out that audio context has some complicated error modes and it will fail if you use it under strange conditions. And so just doing this means our tests cannot fail using those modes and we’re going to hit strange errors in production that our tests can’t fix. And so the question is then if you can cheat on automated checks if you know and even in good faith ways as well. The the AI

7:25:19 isn’t trying to write bad tests here. It’s just taking our instructions and writing tests that are too tied into the structure instead of actually executing code. So how do we make automated checks harder to cheat? If we can do that, then we can increase the quality of those checks, which means which increases our quality bar. And the first thing I really like about this is codebased design. So you can actually design your way out of these bad checks. So what does good codebased

7:25:49 design look like? I’ve talked about this before in previous AI engineer talks I’ve given, which are deep modules. These are modules that hide complex behavior behind simple interfaces. This is a John Asterout idea from the philosophy of software design. If we look at these two modules, we’ve got A which has a large implementation hiding behind a tiny little interface at the top. Okay? And then B is a large interface, lots of functions you can call and those functions individually don’t do very much. Does that make sense? Yeah. Now, if you have a deep

7:26:23 module like a here, you’re going to have fewer structure sensitive tests because you’re hiding more of the implementation behind that interface. If it’s just testing at that interface, you’re going to get better tests. And so, your job here is to force the agent to use that little interface instead of reaching into the implementation to test these weird implementation details. So, I’ve got a skill for this. You can have like the weirdest vibecoded like codebase, the crappiest codebase that

7:26:54 you’ve ever set your eyes on and you can run this skill on it and it will make it better. What this does is essentially gives you opportunities for deepening modules and kind of looks like this. Raise your hands if you’ve used this skill. By the way, not sure how many of my folks are in this room. Yeah. Okay. It’s really freaking nice. Essentially gives you a HTML document. I’ll get out of the way here of all of the different um potential opportunities. it sees. So we can see here we have a before and we have an after where we’re sort of like reducing duplication. We’re creating a

7:27:24 nice deep testable module and then you can go ahead and implement that. And attached to this, there’s also this kind of language that I’ve put together for describing modules. Because like if you ever try and read up about how to structure a codebase, you’re going to find 20 different approaches and they’re all going to be called DDD. And like what you need is a consistent language that you can use in your team to talk about this stuff. And so I have a little codebased design skill that defines what locality is, defines what leverage is, defines what a seam is. I was using

7:27:55 seams before they were cool. And what locality means is kind of how uh well located together all of the code is. How can you change like a small change in one module and have it ripple out? And also leverage is what you get when you have a deep module because the caller, the person who’s actually calling that module gets a lot of value of calling a simple function. Both of those are very good in code bases and good for agents too, it turns out. But I’m sort of

7:28:27 describing all of these highluting coding standards. But how do we actually make sure the agent does them right? How do you make sure the agent creates deep modules and creates good tests and doesn’t write these crap tological ones or structure sensitive tests? Well, I do think most people get this wrong. And my first piece of advice is don’t put coding standards in your implement agent. Okay, let me explain this. If you

7:28:59 imagine the implement agent kind of looks like this where this is all of the things the agent needs to be able to do in its single context window. It needs to be able to explore like to look for the code that it’s going to change. It then needs to actually change it. So make the updates to the files in the green and then it needs some budget for actually debugging the thing. So for running those uh automated checks, for actually checking and verifying that it works. Now this is quite a lot of work it turns out. And if you try to impose your

7:29:29 coding standards on it as well, it’s going to perform worse. So implementation is overloaded. That’s the mental model I want you to have. And so can we find a way to impose those coding standards in a way that isn’t so overloaded? Well, this is my effort. This is my code review skill. And it receives a diff and it reads a file inside the repository called coding standards which you can write, you can customize and then it checks if the code follows those standards.

7:30:01 And so if we look at the reviewer agent here, it also crucially runs it in a sub agent. So it’s got its own context window to kind of handle here. It’s got its own budget. This one it needs to do some exploration, right? Because sure it receives the diff so it knows exactly where it’s located where the code is but it should probably do a bit of exploration just so it has the wider context understands the code but it doesn’t need to do any implementation doesn’t need to do any debugging. So while implementation is overloaded

7:30:31 review is actually underloaded. So it doesn’t have that many jobs to do. This means you can pile in a bunch of coding standards to it and it will do a much better job than if you try to do it with implement. So, I think of this, and this is uncomfortable, right? Because we we all want to be able to just get good code out the first time, but I think of this as the twopart process for writing good code, which is implement, you make it work, and then code review, you actually make it good. You impose your coding standards. And for, you know, the retro

7:31:04 developers among us, this is essentially a red green refactor approach. We use one context window to make it okay. Do the red green and then we do another context window to refactor it. That’s at least how it works in my head and it’s been very successful for me. This means that when you have coding standards, you don’t put them in global scope. You don’t put them in agents.mmd because then they sort of drown out your implement. It may read them, it may not. You put them in coding standards.mmd and that way just the code review agent does

7:31:34 it. Now, I’ve got another idea here, which is I’ve been talking to lots of people today. Lots of people saying, you know, I’ve been talking about review and automated review increased, you know, fixing the PR bottleneck. So many folks say, “Oh, yeah, we just use a third party service. We use a cursor bug bot. We use code rabbit or something like that.” I think that I’ve I’ve really tried to make a generic code review skill in the past that finds all the bugs and does security review and that kind of thing. It turns out it’s really

7:32:05 really hard because you either make it too general and it just gives you false positives that aren’t actually relevant to your use case or you make it too specific. You say, “Okay, find all the TypeScript stuff and then Rust people can’t use it.” So, I would say don’t outsource automated review. Build your own. Build up your own coding standards over time. Share them across your team. And if you have an opportunity to impose coding standards, if you’ve got some docs sitting around that no one reads, this is the place to put them in. And

7:32:35 also when you have this automated reviewer, a really natural inclination for lots of people is to say, “Oh yeah, my code review agent, what it does is it reads the code and then it comments on the PR.” So [sighs and gasps] what your code review agent is doing in that case is it’s providing more work for the human reviewer. The human reviewer then has to read all of these verbose comments and figure out, okay, do we implement this? Do we implement that? The reviewer should commit. It should make fixes. So, it should actually change the things

7:33:05 that it finds because then when the human comes around, you’re reviewing a really nice artifact. If it finds anything that it has any questions over, then of course it can comment, but the default should be commits. Stop trying to oneshot good code. Stop trying to make the implementer agent the only thing that you do and go, “Okay, I’m going to force it to be amazing.” It takes a little bit of, you know, thinking your way out of there, but once you realize it, it is fabulous. So, okay, with all of that process, we’ve

7:33:36 run our automated checks. We’ve now run our automated review to make sure the automated checks aren’t lying. How do we then maximize the PR’s quality in terms of human review? How do we get it like working the best it can? So, we need a human friendly PR. And this is a new skill uh coming into the repo which is currently in progress, but I’ll be releasing it soon, which is the PR skill. This is one I’ve been mulling over for a long, long time. I haven’t quite figured out what the state-of-the-art is. And I realized the best way to make a PR skill is just to

7:34:07 steal all the best ideas that everyone’s got. And it’s very nice. Now, what does a good PR body look like? How would you recreate this skill on your own? Well, the first principle is some reviews are more important than others. Not every review is essential, right? And once you understand this, you realize, okay, that means I can focus my energy on the really important reviews. But how do we categorize that? Well, you got to think about the PR as using this

7:34:39 AWS terminology, which is is it a one-way door or is it a two-way door? Now, most PRs that you have will be two-way doors. You can merge the PR and then always pull it back later. It’s the glorious benefit of being a software engineer as opposed to a civil engineer. Right? Mostly when you’re a civil engineer, it’s a it’s a one-way door, right? If you get something wrong, that bridge is going down. But if you have a two-way door, it means that you can easily revert the change. Now, that might be more um

7:35:10 bit more nuanced than you might expect. It might be that a very simple change accidentally blasts out an email to 60,000 people or something in which case that is a one-way door. You want to review that very very carefully. Involves expensive migrations or data loss. That is a one-way door. Review the hell out of that PR. But also tied on to that we need to understand the blast radius of this PR. What can go wrong? And if things do go wrong, how bad is it? And this means I end up with a nice little sort of summary right at the bottom of all of my PRs, which is the

7:35:41 merge danger. I can see this one is a two-way door. It blast radius is localized. And so I can see, fantastic. I don’t need to pay that much attention to this. I’m just going to sort of review it a little. That’s really important. Next, we need to understand what the PR is even doing, right? And I’ve tried lots of different ways of figuring this out, and the best thing I’ve come up with is using pseudo code. Now, a huge um point of gratitude here to the show me skill from the human layer skills

7:36:11 repo by Dex Holley. This is a phenomenal skill that just essentially dispenses with most text and shows things to you in images and diagrams instead. This makes it a lot easier to grasp actually what’s changing and why it’s changing. So you get the kind of standard sort of set of like mermaid diagrams and UML for you know this is a sort of sequence of things that happened. You also just get these lovely simple ones like this for instance. We can look at this and go okay we’re working in a CLI. We can see a new command has been added and we’ve

7:36:42 got two new things little flags up the top here. Just little summaries like this. It really does make a massive difference. It’s hard to overstate. So you’re trying to like make understanding the why as fast as possible. And I think a third principle here is something you should be thinking about whenever you do human review because because we’re not doing like um because our processes now are so sort

7:37:13 of streamlined and all we’re all collaborating around these same skill files, these same steering files. We’re all building an environment for our agents to operate in together. You should think of the process that produces your code as just as important as the code itself. In other words, when you do a human review, you’re not just reviewing the code, you’re reviewing the system that creates it. And the theory here is that you never want to write the same comment twice, right? You never want to catch the agent doing the same thing over two PRs. And so what’s the

7:37:46 mechanism by which you can make your human review matter? Well, this is a new skill. This is called retro. Retro for retrospective. You essentially take a session that you’ve done. It can either be like a single um agent session or it can be a PR plus the session or you can just get it to look at okay look at all of the PRs that we’ve done over the last week all of the reviews pull them all in let’s do a retrospective on them and it will suggest automated checks and coding standards to make the next one better.

7:38:17 So this is the compounding effect where you essentially by doing human review you’re making the quality of the next human review higher and you’re sort of saving less work from yourself next time. And retro is a really smart skill. It adds a bunch of stuff. So obviously it suggests automated checks. It suggests updates to coding standards.mmd. It does other smart stuff too. So it looks at navigation pointers. How easily did the agent find its information? Can we provide a pointer inside agents.mmd to help it out next time? It looks at tool economy. Are

7:38:49 there different tools that we’re using in the session that can, you know, could be made more token efficient? It’s amazing how many things this catches actually because those are often really hard to debug from the outside. It just looks at bloat as well. So, are there bloated steering files? Are there bloated skills that contribute to these bad results? Can we make them more organized? So that’s the goal is to make human review faster. We do that by layering up automated checks. We’re layering up automated review. And we make the human

7:39:20 review as painless, as simple, and as kind of optional as we need it to. You really don’t need to review every single two-way door. Every single one-way door you do. And so these are my skills. AI her.dev/skills. I’m going to be shipping version 1.3 this week. It has been glorious hanging out with you. It’s been a really nice conference. I’m going to be outside in the lobby if anyone wants to have a chat. Uh thank you so much for having me. Thank you, Paris. [applause]

7:39:50 [cheering] [applause] >> Thank you so much. >> All right, let’s give it up for Matt once again. [applause] I love that retro skill. I think I’m going to use it very very soon. All right. Okay, so up next is somebody from Deep Mind U. So we have Martin who’s going to talk to us about Gemma. He’s going to talk to us about some architectural fundamentals, but also

7:40:21 what makes uh Gemma so fast and unique. So please let’s give it up for M for Martin.

7:41:18 No, I’m not seeing anything.

7:41:50 Yay, it works. Today I get to talk about something that I think is a very central component of releasing open models and that’s about efficiency. Because the moment you release a model out into the wild and people can actually use them, uh that’s the moment that they become constrained. Constrained because you know they’re being run on your mobile devices on your laptops and I can kind of assume that there are not data centers that are running these models. So efficiency

7:42:21 becomes a very big part of it. And I would like to focus on a couple of perspectives on efficiency. knowledge, size, speed, the complexity of the architecture, but also the architecture in itself. And I want to dive deep into some of these components and see what we can learn from it. One part is per layer embeddings. It’s a very big thing in the smaller models of Gemma 4. There’s these two billion and four billion parameter models and they use per layer embeddings

7:42:52 heavily. Now they essentially explain what that E actually means in E2B and E4B. We know the A right in mix of experts models they stand for active. But what does the E then stand for? Well, let me show you. And this is the architecture that Gemma 4 generally uses, right? It has local tension, has global tension, group query tension, there’s some normalizations here and there. And then right at the end with the smaller models there are suddenly per layer embeddings there. And as the

7:43:25 name implies these are embeddings per tokens but even per layer. So they’re much like the token embeddings that you know we all know so well but this time they’re being added with every layer. That means that the token high has a very different embedding on layer 1 than it has for instance on layer four or layer 10. It’s a way to add knowledge to the model without having to, you know, store them in memory. During this processing, the model can

7:43:56 actually weigh how these tokens uh are being used. So for a given context, the model might say, “Okay, I want to focus a little bit more on this aspect of the embedding rather than that aspect.” And now the great thing about using something like this is that it’s a lookup table of information about the tokens. And when you have a lookup table, you don’t need to store it in VRAM. You don’t need to store it in RAM. You can just store it on whatever storage you have. It’s a rackl like behavior that’s happening here because during inference, we only need to load

7:44:27 in the tokens that are actually needed. And so that explains that the E actually stands for effective because that per layer embedding lookup tables, billions of parameters, right? But they’re not being used during inference. we can separate in a way part of the knowledge from actual model capabilities and this this this idea of separating them is becoming more and more pronounced in the field of open models. So we have knowledge you can do knowledge efficiently. What about size for

7:44:58 instance uh you can create a very small model but you still want to have a lot of capabilities. So what you can do is you can do quantization aware training. you reduce the precision to something smaller while trying to maintain the capabilities as much as you possibly can. And that’s a very nice way to approach this. But we can actually take it a step further with mobile quants. And that’s a very specific schema that was developed for these two smaller models because when you have smaller models and you’re going to quantize them

7:45:29 to even a smaller size, you have to make sure that you don’t lose too much performance. So a very specific schema and looks a little bit like this. So we have the token embedding layer right you can actually reduce that all the way to two bits and retain a lot of the capabilities of the model still uh and now the reason I will come into a little bit later for global attention four bits 4bit is what we generally use right it’s a it’s a nice balance between the size of the model and capabilities

7:46:00 but for local attention which you know zooms in on the thing that we want to know more about it helps to have a little bit of a higher precision Especially with the KV cache feed forward network is four bits but then the per layer embeddings that we just talked about those are two bits because they kind of compensate for the lower position of the token embeddings as well. And so when you combine all of this you have a nice schema for a specific model. And what you see is that making sure that it’s efficient also

7:46:33 means doing a lot of customizations to these models especially at sizes like two billion four billion parameters. And with all of that you have knowledge, you have efficiency in size. What about speeds? Well, with speeds we use something that’s becoming more and more popular fortunately these days speculative decoding with multi-token prediction. And it works by using a rather small model to kind of suggest tokens that the bigger model, the model

7:47:04 you want to actually run, can use because that allows the bigger model to then only have to validate all of the tokens that it sees without having to process all of those tokens themselves at once. So we have a target model, right? It’s a big one. It can be whatever architecture that you have and a draft model, a smaller one. And with Gemma 4, what we aim to do is do some KVK cache sharing. Why? Because that simplifies the process for the smaller model, the candidate model or

7:47:35 the draft model much, much more. Because what happens is that the target model takes in and query, does a single round of processing, one single pass, generates its KV cache, and then shares it with the smaller model. And the smaller model can then use the output of that and continue on processing and generating tokens. It will take the input, generate one, two, three, perhaps eight tokens for the larger model then to validate. And so the larger model

7:48:06 only has to look at all of these tokens at once and decide, okay, I like these tokens, but I don’t like so much these tokens. So I’m only going to accept these first two, for instance. But it’s still a single pass, right? So all it needs to do is then add another token because it can it did the computation for the entire sequence anyway and then it generates three tokens almost at the cost of a single one and that speeds things up tremendously.

7:48:38 Now obviously nothing is a free lunch right? So there are pros and cons to a method like this. You add a model it’s a very small model but you add it anyway. It also differs greatly on the use case, the speed up that you get from something like this. The thing is some tokens are super easy to predict, especially in structured tasks like code for instance. But in creative tasks, it’s very hard for the smaller model to really figure

7:49:08 out okay, what kind of tokens am I going to suggest? So more of these tokens will then be rejected. Well fortunately there are more and more coding task more and more structured task that we see out there. So for those use cases multi-token prediction works exceptionally well and then you have efficiency on let’s see knowledge size speed uh what about complexity? Oh did I see this one here? Let me remove that

7:49:39 server. And then we have complexity because that’s also a thing, right? If you have a model that’s not open, it really doesn’t matter that much how complex it is for the end user. But when you release a model out into the wild for folks to use, uh, let’s make sure it’s not too difficult to actually, you know, understand, use, uh, and process. And so, as a consequence of that, there was this model that was released called an encoderree model. And the encoder free model kind of looked at the encoders for you know processing these

7:50:12 multimodal inputs and said that’s a lot of work. What if we remove them? For many of these models in the Gemma 4 line you have an audio tokenizer an audio encoder and the same for images. And that’s great. That works really well. It’s a staple in the field right to have these encoders for different multimodal entities but they still take up a reasonable size. It reduces time to first token and it’s something else to consider when for example fine-tuning your model

7:50:44 with the encoder free model. They try to just remove them entirely and put a lot of the burden of processing these multimodal inputs onto the model rather than these encoders. For audio, it works a little bit like this. You have an input, a sequence of amplitude values and the the only thing that happens is they cut it up into pieces, make it a sequence of well audio tokens and then project it onto the dimensionality of the model

7:51:14 itself and that’s it. So the only parameters that you have are kind of in the linear projection, but that’s about it. And then all of the contextual stuff that was happening in the audio encoder before is now being handled by the LLM itself. So you’re kind of shifting all of these parameters from the encoder to the LLM. The same can be done with the image encoder. The one problem though is with images is 3D information. We can’t just cut it up

7:51:45 into pieces, make a sequence out of it, and just hope the model processes it well because it doesn’t. I I can promise you that. So what we need for that are positional information. So instead of doing a a direct projection from the patches onto the model, we take those patches and provide it with additional positional information. So it knows that well let’s say patch four that small image token um has actually some meaning to it because you can imagine if you have a very wide image position four

7:52:17 means something very differently than if you have very tall image. And so what happens is you just use the positional embeddings uh projected and what you have again is an encoder that’s entirely removed. What’s so nice about something like this is that it reduces time to first token that very much helps. you put all of the capabilities onto the model and you reduce uh a big portion of the parameters because especially at smaller

7:52:48 sizes something like 300 million parameters or 500 million parameters it doesn’t sound large when we’re talking about hundreds of billions of mod of parameters for some of these models but at two billion sizes or 12 billion this makes a big big difference especially if you can use it for the capabilities of the And so we have knowledge, we have size, we have speed, we have complexity, all different ways to focus on efficiency. And there was one last thing that I

7:53:18 really wanted to focus on and you know explain a little bit more about what it means and how it works and that’s architecture. Regular models, regular LLMs are all memory bound and those have advantages and disadvantages and they’re great. They’re used for many different purposes and and many different fields. What if we make them computebound instead? What if we flip the narrative 180 degrees and try to approach it from

7:53:49 a very different perspective? And so what you get is not auto regression, it’s something entirely different. Diffusion. Can we use diffusion for LLMs and still have it being meaningful? have it used in a way that you know a lot of people can still use it but for very different use cases. Now let me go through a little bit about about what diffusion means in large language models uh especially because we will come to a subject that I think a lot of folks will have seen in the last couple of weeks.

7:54:22 You start with an input, right? We have a query, an input that’s being processed. And normally one model would do that and now kind of the same model does it. But what happens in the fusion, we have something called an encoder. And the encoder in the fusion gemma at least is still just a decoder with causal attention. It’s a fine-tuned Gemma 4 model. No crazy different architecture or what have you. It’s, you know, one of the models you saw before. The reason why it’s called an encoder though

7:54:53 because it’s meant for processing the prompt and so it generates a KV cache the context that we need to then build upon and slowly and more surely do inference. So we start with the KV cache but do to do the actual processing to actually generate tokens in the fusion you don’t start from scratch. You don’t just you know here’s one token and here’s the second and here’s the third. Now you start with a noisy con canvas. It’s typically sized around 256

7:55:24 tokens and then the job of the den noiser will be to actually process that and iteratively fill it up with tokens that it’s very confident about. It is the exact same model though. It uses the exact same tight weights gem for 26B model. There’s one difference though. If you have a large canvas and you want to update the tokens simultaneously at the beginning and at the end, you kind of need a different type of attention. So when it’s in the

7:55:55 denoiser mode, it uses birectional attention instead. So it can lock both directions. What happens after that is it gets the KV cache of the encoder because again it was meant to process the state and get it ready for the deninoiser to do its work. And then the deninoiser can well predict or suggest what tokens should go at which specific places because we have this initial canvas of tokens and all it needs to do is update them at the appropriate places. So there’s this

7:56:26 predicted canvas that it then has but you know it doesn’t do stuff in uh perfectly at once. So it needs a couple of steps to do that. There are some tokens it’s very confident about obviously and some not so much. So what you then get is in so-called accepted canvas some tokens are accepted and some tokens are not. The tokens that are not accepted though those are renoised because the entire purpose of this model is to dn noiseise it to remove the noise from the original canvas and find the

7:56:58 actual tokens that should be there. And so we end up with a renoised canvas and it go can go ahead and do its second step. And now in the second step it again uses the KV cache and it can use this KV cache to again do a similar processing. But what we found if if you were to do just this, it’s not a perfect way to approach this. Because in the very first step, it had an initial idea of what it wanted to do. Even though some of the tokens weren’t accepted, it

7:57:30 still had a direction it wanted to go into. And so there’s kind of a skip connection there, a self conditioning where you take the logit from the previous one and feed it to the input of the second step. And then you can do this process over and over again, typically eight steps or something where it updates this canvas. And then in those eight steps, sometimes less, sometimes more, you get 256 tokens. And it makes your model go incredibly fast when served for a single

7:58:01 user on a larger GPU. Efficiency takes a very different meaning than when you compare computebound to memory bound. But there’s still many use cases for that. One we didn’t quite expect would blow up in the last week or so, which is obviously Jeff. because Jeff being a a a foundational decision model, you know, essentially a foundational classifier that’s being used to make decisions happens to work

7:58:32 quite well with diffusion. And it works a little bit like this. So let’s say you have a situation, the printer is on fire. anything that requires you to want to take an action. You describe the state, the situation, what have you, you again take that encoder and you process it for the purpose of creating a KV cache. What then happens is instead of having a fully noisy canvas, you kind of prefill some of these tokens and the prefill are

7:59:04 then related to the actions that you want the model to make a suggestion on. Okay, the printer is on fire and your options are to either, you know, decide if I should evacuate or call it. Those tokens don’t change, but the others are still noisy. That’s the one it needs to fill in. And what happens is that when you do that, it can make a prediction for only those two tokens and using the things that we saw before, make a probability of whether it should

7:59:35 do something or shouldn’t do something. just one den noising step. Technically it can do more but if you want to make fast quick decisions uh this is all it takes to create well apparently it’s called diffusion gemma Jeff now but that works extremely well because nobody fine-tuned that model the only thing that happens is just limit it in a way but that showcases also this concept of efficiency because it’s okay to limit certain things in certain ways because

8:00:07 it opens up the way to make sure it can be use for things we didn’t imagine before. When you combine all of that together, one thing that makes all of this happen is that it’s all about the community, right? Efficiency is not for us, it’s for you. It’s for the folks that use actually these models. And whether that’s on mobile, on a laptop, on bigger GPU, that doesn’t matter. when you release these open models and you’ve seen a bunch of talks already today

8:00:38 talking about that it needs to be useful for you. That’s the main purpose of trying to give back to the community. But in the same way when you do that you can see the community giving back to you because a decision model based on a model that wasn’t necessarily intended for that. Well, that’s very cool to see that gives a lot of innovation from a field uh that you know we didn’t do but by giving people the option and the

8:01:10 opportunity to play around with these models to open them up to see what’s happening inside them to tweak it to change it you get these fun cool models that who knows where it’s going. I have no clue. uh it’s now a big hype. I think for a good reason, but the thing is with hypes, we’ll see how far it gets. And when you then talk about efficiency, there’s all these different types of fields and perspectives that you can focus on. And I could stand here and

8:01:40 talk for hours upon hours on what efficiency means alongside performance. And that’s not an easy thing to do, right? There’s so many of of these incredible open models out there that try to focus on these things because it’s not just about performance but also of on whether you can actually run the model because it doesn’t matter how good a model is if you can’t run it it’s meaningless and so a lot of attention from different parties and you see a lot happening in Jama

8:02:14 devices and hopefully this helps you get an idea of what it means to focus focus on these on these smaller models. It’s a very different narrative than going bigger and bigger and bigger. That’s also interesting. Gives a lot of very cool capabilities. But these open models, these small models, especially when they’re on device, on edge, your small devices, they need to be capable and they need to be quick. And efficiency from many different perspectives is a very big

8:02:44 part of that. Thank you very much. All >> [applause] >> right. Oops. Okay.

8:03:30 All right. Thank you guys. Let’s give it up one more time for Martin. [applause] All right. So our next speaker is going to come in a few minutes. In the meantime, I have so many questions for you guys. So um I just would like to know what was your favorite part of this conference? Like does anybody want to share?

8:04:04 Okay, maybe two open-ended questions don’t work in a crowd big crowd like this. Okay. So, um uh you guys I I don’t know if you guys went to uh to the expo, you checked the swag, you checked like uh but but I like I just want to know for example for me what I like the most is uh seeing people at the expo discussing and and and making new friends. I think it’s something that is uh quite obvious to say right like every time we go to a new place, every time we go to an event is to meet people. But I really like that it really becomes like

8:04:35 a family. I’ve been um uh attending AI engineer for quite a while now uh around the world. So in San Francisco, in New York, London, and here. And it’s so cool to be able to see like sometimes we see the same faces over and over and then we make new connections. So my my advice to be honest with you guys I know we we don’t have a lot of time left but yeah just make sure that you connect make make sure that uh you you collaborate because I think that the through this

8:05:05 community like a lot can happen and you never know it doesn’t have to happen now it can happen in the future. Um, another nice moment that I saw is uh so many people lining up for to talk to uh to our speakers like yeah so many people come to me and they ask me also about um some some of our speakers apparently like yeah um P.A.I. was quite popular. Uh ARM also is quite popular. So it’s good to see that uh our audience is uh is quite diverse. Nobody

8:05:36 wants to share what their favorite moment was. the food. All right. Yeah, I I guess so. Yeah, I guess so. Yeah, thank you Sarah again [laughter] for sponsoring for the food. So, apparently we nailed that. Okay. Any any talk favorite talk? Gemma 4 was great. Okay. Wow. Nice. What else? Okay, that was Gemma 4 and that’s it.

8:06:07 Like we should have just have gym four for the entire conference and that’s it. The kids that was hilarious and I love Kitza. Yeah, shout out to KZA. But but yeah, he’s uh he’s hilarious. Yeah, I agree agree with that one. Um all right. Okay, so enough of me, but this is our last presentation of the day. Okay, so thank you so much for hanging in there. Um but before we get started I would like to introduce introduce uh

8:06:37 Olivia Tibu from Gradium who is going to talk about the missing layer of conversational [music] AI. Let’s give it up for Olivia. Okay. All right.

8:07:11 You’re extra. Oh, it’s not what I meant. It’s not what I meant either. All right. should get started. Let me see if that

8:07:41 is working. That is working. Cool. All right. Should I get started? Yes. All right. So, let’s get started. Hi everyone. My name is Olivia Tubul. Uh I’m a CTO and co-founder at Gradium and today I’m going to talk about the missing layers of conversational AI. It’s a mysterous title. So, we’ll see what that mean. Uh maybe before we do, let me introduce you uh to who we are at Gradium. So we are a one-year-old startup

8:08:14 uh focused on training uh voice models. So those acronyms you might have seen if you don’t did not ST means speech to text, TTS text to speech and S2S speech to speech. So all these kind of flavors of models uh we train and serve at Gradium. Maybe before that if I go back to to the genesis of um of Gradium is QAI. So

8:08:44 maybe you’ve heard of it. I don’t know. Yes, some of you. Great. So CQA is a a Paris-based open science lab that was founded in 2023 with quite some money and a lot of freedom and out of this freedom they you know made some great scientific breakthrough. uh Moshi is probably the most famous and I’ll go back to that a bit later in the presentation uh and of obviously some others but that um I would say the success of those voice models were so

8:09:15 big that somehow we thought in order to get the you know bigger reach of our research idea we should turn that into products. So the goal of Gradium was really to bridge this gap from research to product and and here we are one year later. So, conversational AI, right? Uh where where do we got that? Actually, today pretty much everywhere you’ve heard about voice agents, whether it’s in gaming, live

8:09:46 streaming, assistance, robotics, wherever you want to talk to a machine, there is a voice agent involved. And I would like to explain you a bit of the history uh today, not only but part of it. And if I go back to well what I could call provocatively the prehistoric time of voice agents meaning before LLMs uh you probably remember that right I think that was the closest

8:10:18 um I would say glance at AGI back in the days right it really felt like AGI right that was the iPhone and Siri and Siri was nothing really agi was A very nice uh product, engineering product made of different layers, lots of layers, six of them, sorry. First one is P uh the speech to text the speech recognition. After the audio is turned into text, it was turned into you

8:10:50 know classification classifiers trying to say what is the intent, what is the object, what is the subject, what is the complement, stuff like that. a dialog manager, uh, some calls to APIs like, you know, check the weather, check the calendar on the phone, stuff like that. Um, [snorts] probably the most, you know, I would imagine template and regex’s intensive part was to generate some text uh for Siri to be spoken out and then the TTS

8:11:20 would speak it out loud, right? Then what came after was LLMs and I say maybe we don’t need all of that, right? This natural language understanding, those templates all gone. LLM, there’s a bit of a lion here, right? There is still a dialogue manager in modern uh voice agents. And those LLMs, they become stronger, right? And they are now able to connect to the external world. They can interact. They

8:11:53 can talk to a database, a web search, MCPs, tool calls, retrieval, rag and they became voice agents and that’s what you’re seeing here is actually how I would say 99% of voice agent are working. It’s called the cascaded model ASR or TT ST LLM TTS right they are not trained jointly. Those are three pieces of AI modules that we plug one into the other.

8:12:26 There is a an extension of this which is okay maybe we don’t need three pieces. Maybe you can just one big model and that’s we call speech to speech and then [snorts] there is no turn taking to take into account. The model itself knows where to speak and when to listen. So this is to give you an overview of what is convers conversational AI. Uh so I can also give you a glimpse of what we do and then I’ll talk about the engineering aspect and the AI aspect of

8:12:58 what’s under the hood. So what do we do at Gradium? We basically do all of this, right? We do the text to speech, the speech to text, we do the text to speech on device. Uh we do some speechtoech translation. We do some voice design because when you want to say something out loud, you want to say it in some voice. Can be a French accent voice for example or it can be a female voice, high pitch, low pitch, whatever. And coming soon is actually the speech to

8:13:30 speech. So this talk is called uh the missing layer of voice AI. So what is actually missing? It feels like it’s it’s complete, right? And some people will try to convince you this picture is complete but it’s not. And those are just some examples. It’s not everything. Uh but we’re going to talk more about that. So if you put you know voice AI in the wild, well most likely it’s going to go wild, right? Because most of the um the applications

8:14:02 do there are more on the phone, one speaker uh you know one computer talking to one another. But put that in real life you get like many speakers. it confuses the agent. Noises confusing the agent long sessions you leave the lose the context and also the real global scale. Uh and what I mean real global scale is like not the ability for anyone to reach your your service but like the ability as that was Siri was doing like everyone on his phone can talk to it

8:14:34 instead of typing. So underlying those issues are some challenges and I’m going to pick two of them to explain to dig into how how we do and what’s the challenge. So if you want to do um voice models you need to several things. Uh basically you want something that’s fast uh and something you can serve at scale. And so you have

8:15:04 to think about this fast and at scale this real time at a scale from from the ground up. So it means like designing a model that will be able to scale and that will be able to be to serve fast optimizing the inference and having a serving infrastructure that will support it. I’m going to talk about the first one and the last one. So how do those how does model work? Um, so when you want to design a new model, you don’t start from scratch

8:15:36 basically, right? You start from something that’s similar and try to adapt. So what you really need for and I’m talking about the speechtoech uh system you need something that will take the speech as it comes and that will be able to take more and more speech as the conversation keep going and that will be able to as soon as possible start speaking in return right and of course there is a one of the requirements that it’s speech in speech out right so even

8:16:10 if we don’t those models. There are a family of models that is really similar to this and that satisfies two out of those three requirements which are LMS. LMS you take a text in and a text out. So how do they work? You have a context. At the beginning it’s the prompt but then it’s a context. Those are words and the LLM is trained to predict what’s next. Okay. So in that situation gradient is an AI model company based

8:16:41 what’s next is it in off at blue bird you know it gives a probability to all of them so of course you know in seems to be a good candidate so if you represent it that way at the bottom line this is the context on the top line this is the prediction what’s interesting is that and that’s why it’s called auto reggressive models the next prediction is going to be part of the context at the next round. So you say oop sorry um gradium is unpredicted

8:17:14 AI AI becomes part of the context and now it can bring a new one and another one and another one and this way on the go you will uh predict and generate a full sentence. So can it work on audio? Feels like it’s the same right? Audio is a sequence sequence to sequence. Why could not it work? Well, there is a dimensionality issue here. Um, the problem with audio is that this text, Grandom is an IMO company based in Paris. It’s nine words. So, like eight words for the context one

8:17:45 you’re going to predict. It’s 3 seconds when it’s spoken out loud. But 3 seconds of audio, if you represent it on a computer on a normal quality, it will be like 24,000 samples per second. those eight words becomes seven 72,000 sample uh values. So it’s much longer of a sequence. So you have 10,000 times more sequ uh values to predict and since the transformer architecture

8:18:17 which what’s the LLM is relying upon is relying on self attention which is quadratic. then your uh attention matrix is 100 million times bigger. So it’s not tractable. So the first trick is to turn the audio into tokens. And that’s the neural codec. And actually it turns out you can do that. You can train a model to compress the audio in tokens from 24

8:18:47 kHz to 12 hertz 12.5. So you divide by 2,00 then you have a tractable number of tokens that’s a sequence and you can start predicting because at the end of the story will decode them into back to audio. So if you wanted to do that to train a dialogue system you could say me I’m the I’m the LLM okay I’m the gray one. So, we have a sequence of me talking, me talking, me talking, you talking, you talking, me talking, blah, blah, blah.

8:19:18 That’s a sequence and we’re going to predict the next audio tokens, right? But this, sorry, sorry, sorry, sorry. This doesn’t really work like this because in a conversation, it’s not like I talk or you talk. We might be talking at the same time. Otherwise, it’s just like walkie-talkie uh conversation, which which is lame. So in practice you have to invent something new that we don’t have in LLMs is multiream hierarchical transformers. So like now you don’t have a single line

8:19:48 of time for the context. You have two you have me speaking and you speaking and those two things will attend to to be able to predict what’s next. And so those models they can listen and talk at the same time. Give you an example. Sorry, of course.

8:20:23 So, the planet is Sirius 22. Can you plot a trajectory course to it, please? >> Yes, sir. >> Okay. How long is it going to take us to get there? >> I’ve mapped it out. It’s approximately 5 months to get there. Okay, that’s that’s not too bad. Uh, do you think we have all we need on board the ship to start the mission? >> Yes, sir. We have everything we need. >> Okay. >> Today, >> okay, I’m going to skip the second one. So, you see like it’s very natural that the model is trained to listen and talk at the same time. So, there is no latency whatsoever. And that was Alex, our chief scientific officer in 2024.

8:20:57 So now you’re there is something I didn’t tell you is that in 2024 we had this technology at QI but we’re not selling it. So what’s the missing layer here? Well, [snorts] there is something here is that this one model is just one model and it’s capable of um listening, speaking and you know processing language. That’s a lot of things for a very small model. So when you talk to it at some point you will think okay it’s not as good as you know

8:21:29 the limitation is even worse is that this model is kind of airgapped it doesn’t talk to the external world. So if it doesn’t talk to the world if you cannot act on you know booking an appointment or something in production it’s it’s just a gadget. So the big challenge for speech to speech today is to be able to do seamless integration with tool calling. And if you do that, then you have the same experience while being able to do what a voice agent should do. Okay,

8:22:00 so that was about how we build a model roughly, not all the recipe. Um, and now let’s say you have the model, you want to scale it and serve it to the world. How do you do that? So here we got the latency problem. So you have one person with his computer. I’m going to take the text to speech example here sending some text to our server and we have to answer some audio back and if you want to plug

8:22:31 that into a voice agent it has to be fast has to be real time and by real time we mean less than 300 milliseconds. So that’s the latency budget that you have for the whole thing. If you don’t, then it it’s going to be like a weird conversation where you wait for a while before getting an answer and you’re going to be bored. So, it has to be fast. And during that trip, you have actually three layers that you have to go through. One is a transport layer,

8:23:02 right? Network layer. The information has to travel from point A to point B. Then you have the product layer because like someone sends me a request. I should check this is that legitimate. Does this person uh does this person have the permission the credits? He wants this voice. Do we have this voice? Where is it? And so on and so forth. So you have to enhance the request or deny the request. And then there is a compute.

8:23:33 So how do we do that fast? So first problem you have to solve is a latency problem. So let’s say I’ve put my server somewhere in Europe and my client is somewhere in the US. Well, just by going round trip through the Atlantic Ocean, I kind of I’m wasting 100 milliseconds, onethird of my budget for just transporting the information. So how do I fix it? Okay, easy. I’m going to put servers everywhere, right? It’s more

8:24:05 expensive first, but I’m going to reduce the trip from the client to the server. But now I got new issues, right? Because I had the system with no replication and no distribution. Now I have a distributed systems where the information has to be synchronized. Of course, if you want to go fast, you put caches, right? So I put I have a database. I have cache of my database. It’s already an issue per se for you know all the cache invalidation. But now you have many caches that has all to be synchronized alto together.

8:24:36 Otherwise, you create an API key and you’re going to California. It doesn’t know about the API key. That’s weird. Okay. So, you’re trading latency for complexity, engineering complexity. We can handle it, but still it’s it’s important to be aware of it. Then the second thing which I think is very very new. You know when we started a year ago, we were thinking there is no way if I need a new GPU to handle my request. There is

8:25:07 no way I can ask my cloud provider for a GPU. He gives me a GPU or she give me a GPU. Uh the GPU is ready and I pull the weights. I pull the Docker image. I warm up the GPU and this going to happen in 300 millconds. Impossible, right? like if you try you you you can maybe make it minutes right today what’s happening so we we never put that as a as an option you have to predict and to be ready for the load but

8:25:38 now what’s even worse is that the GPU scarcity is so important that you may not be able to have it even like in next day so maybe you want one more GPU but you don’t have this GPU because there is no GPU available in your region because one of the cloud provider are saying sorry there is everyone wants a GPU we don’t have it so how do you handle that so again you’re going to trade you’re going to mitigate this risk with more complexity and there are two things you

8:26:09 can do the first thing is I’m going to go to different providers there is more the odds the odds to find the GPU is is better or I’m going to try to uh have my model run on different architecture on different models on different chip providers, right? Not only Nvidia or not only H100 or stuff like that. So that I increases the odds of finding a GPU. So again to mitigate that risk, I’m increasing the complexity of my system.

8:26:40 And how do you handle like burst, right? Because you can uh sometimes everybody wants at the same time uh to be served, right? And there are different strategies you can do and we take two options. First one is to say now let’s take advantage of all those clusters and reroute the traffic from one to the other the closest one. Or you can also do change the batch size of your model. Maybe you can trade a little bit of latency with a bigger batch size right

8:27:12 for the time where you hit the wave. You increase the batch size and then you decrease it. In practice, if you put all that one after the other, a good model, fast inference, fastly served, that’s what you get. So, we did um we’re building this week a new beta model for text to speech and we did that race where we click somehow at the same time, but we measure the time for uh how much it takes to get the audio back for us and

8:27:44 some providers. And this is what we we got. >> Hi, I’m Gradium’s newest model and I’m the fastest text to speech we’ve ever built. Faster first audio, more natural conversations, no waiting, and guaranteed speed. Try it today. >> Yeah. So, we’re happy that we’re, you know, pushing the boundaries of of latency. you know, the best models were about like 150 millconds and we made it like three times faster. So, at some

8:28:16 point there is no there is no limit, right? It’s not never going to be, you know, one zero milliseconds, but we’re happy that we’re able to put all that together uh to make it happen. Now, to close this talk, I’m going to take talk about can we go further? Actually, I’m telling I told you we cannot go zero millisecond and that’s true. But can we go zero milliseconds of travel? Can we go zero milliseconds of product and still have something fast that would scale even

8:28:47 more? And that’s the idea of moving the compute from the server to the client directly. And for this you have to have very tiny models that can run on your device on your phone on your brother on a raspberry whatever. Uh so we also released like grad phon which is the smallest it’s not exactly the smallest one of the smallest but with a very low u number of parameter 100 million parameter um only uh and a

8:29:20 very good accuracy and this you can also try feel free to reach out if you want to try those models I’m going to give you a little bit of examples of what it sounds >> only 100 million parameters and I can speak in any voice you give me. >> And here’s an example with a brand new voice cloned from a 10-second sample. >> Okay, we’re going to skip. >> This voice didn’t exist a minute ago. I generated it from a single 10-second recording.

8:29:51 >> Same model, completely different voice. >> Cloned on the fly from a short clip. >> All right, so that’s that’s pretty much all I wanted to uh tell you about. you know what’s below the hood in in the voice uh AI how we build conversational agents how we train those model how we serve those models and how if we you know put together nice research and nice engineering we can really uh push the boundaries of of conversational AI thank

8:30:23 you [applause] [applause] >> thank you so much Olivier let’s give it up for Olivia for and Gradium Nice. [cheering] Oh, all right. >> Okay. So, if you follow the QR code that was uh that was shared with you guys, then you can have uh credits, right, to to try out Gradium. All right. Thank you. Okay. So, uh we still have a few

8:30:54 more things for you guys. So, don’t don’t go don’t leave just yet. Um actually I’ve been working in AI for uh quite some time now. I can’t say that it was a long time but it feels like a long time. And one of the questions that I get the most is what do you think is going to happen to the next generation? And it’s a tough question right because things are moving so fast and it’s already quite hard for us as adults to understand the impact of all this technological change. Um, but luckily for us, there are people that are doing

8:31:24 something about it. And uh, I I would like to welcome Cassandra Chin, who’s going to tell you more about it. So, please let’s give it up for Cassandra. >> Hi. So, my name is Cassandra Chin and I am one of the kids workshop instructors for AI Engineer Paris. This weekend we ran a kids event where we educated kids and helped teach them to learn about

8:31:55 technology to prepare them for this AI world. But before I get too much into it, I want to talk a little bit about myself and why I’m so passionate about teaching kids technology. I actually started teaching kids when I was a kid myself. I was 13 years old at the time and I would teach kids workshops over the weekend, Raspberry Pi workshops where kids would get to tinker with different technologies. But even before I started teaching in my

8:32:26 household, there was lots of technology. We had 3D printers, a lot of different technology. I remember building my own computers. I really believe in inspiring kids to enjoy technology rather than pushing them. Because if you show kids that technology is this very fun thing, not just dad or mom at the computer, like show the kids what technology can actually do. That’s what we really tried to do in these kids workshops.

8:32:57 And for the kids workshops this weekend, I taught a workshop teaching kids how to code with AI. The kids used Mistral Vibe to build their own games. And I was really surprised at what the kids could do because we have these preconceived notions about how to code, what you can or can’t do, but the kids, they just have ideas. Some ideas they have are like building an RPG game, a platformer,

8:33:28 a banana game. a lot of creative ideas from kids but they just prompt and the AI could build it for them and with the we had an organization called AI tinkers they brought in a lot of underprivileged kids for uh to the workshop and these kids they they could learn really well like it didn’t matter what background they have and I think it was really great that they also brought in a lot of girls another issue with technology is

8:34:00 there’s a lot of gender biases. As kids get older and older, they start thinking that technology is a guy’s thing because they see more of the industry. But if you catch kids when they’re very young, they don’t have this idea that technology is a guys thing. And a lot of girls can be inspired to join technology like I have. So I think these kids workshops are really great for kids. The workshop which I taught, we had the kids build the games and we actually

8:34:30 connected it to a GitHub repo which had a lot of art assets and a agents.mmd file. So this created a more kids safe way with AI because a huge concern is how is it safe to let kids use AI? But with this agents.mmd file, we were able to instruct the AI to act more friendly, use less tech jargon, and be more helpful, run commands rather than asking you to run commands. A lot of

8:35:01 instructions which make the workshops more kids friendlies. We have this video which we produced about the kids workshop. So if you would please watch it, I think it’s really nice. >> [applause] >> We’re here at AI Engineer Kids Day in Paris teaching the local kids about AI technology, about models, about neural networks. This is a great way to expose kids to AI in a fun way, in an approachable way, and something we love doing at Neo Forj because we’re helping

8:35:33 train the next generation of AI engineers. There’s a whole mix of kids here uh from a bunch of different backgrounds. Uh some of them are already expert coders at 8 years old and some of them are in their teens and they need a little bit extra help. But that’s kind of what excites me is we’re putting them all in one room. There’s peer-to-peer learning. They’re teaching each other and they’re helping each other so that when they grow up and they start working in the industry, they’ll have a handle on their own lives, their own careers, their own

8:36:04 ideas. For my workshop, the kids use Mistro Libe and it is a coding agent which lets them create their own games. It’s really interesting what kind of games the kids build. Some of them choose to build an RPG game. Some do platformers. Another kid did a balloon popping game. And what’s interesting is the there’s no language barrier. Even so

8:36:35 the kids mainly speak French. They can type French to the agent and it fully understands French and still codes the games for them. So I think this was a really good experience for the kids because they got to build their own unique games. So my organization AI tinkers organizes a lot of events and having an event that will bring together the kids and their

8:37:07 family members. So our usual community members that could be the parent or [music] the big brother or big sister. That sounded really exciting so that they can build together and I believe that’s part of community building. And there’s a reason why we were really excited about doing this.

8:37:40 [applause] We are running these kids workshops as a part of the AI engineer conference series. So we will be running the next workshop at AI engineer uh New York and also AI engineer code in San Francisco and next year we’ll be running it at London. So if you’re at those events, please bring your kids. We’ll welcome your kids and teach them.

8:38:10 [applause] >> Thank you. Yeah. Thank you, Cassandra. Let’s give it up one more time. That was awesome. Yeah. Thank you, Cassandra. Thank you, Neil Forj, uh for doing yeah, such a great job. Uh I’m a father myself and I can relate. This is super important work. Um yeah, so thank you one one more time. And uh let’s hear it from you guys as well. Thank you. You you made it. Hey, you made it all the way here, right? I know somebody somebody told me earlier is like, “Hey, you clap too much. My

8:38:41 hands are hurting.” I’m like, “Hey, dude.” Like we gotta find a Oh, you’re there. You [laughter] recognize yourself. Nice. Yeah. Um, but thank you so much for being with us and I would like to uh thank everybody that helped us organize this event, right? Um, I would like to uh sorry, I would like to thank our sponsors, Mrol first for organizing the event and then Nvidia as our uh platinum sponsor, but also all our gold and uh

8:39:11 bronze and and silver sponsors. So, let’s give it up for them, please. >> [applause] >> And also I think you should be proud of yourselves. So I I heard that many of you have traveled all the way here from different places. We heard somebody from French, Polynia and then Australia to live and different parts of Europe as well. I think this is so cool that people come from different parts of the world to come and and and be with us here and uh we really appreciate it. So um yeah. So let’s let’s hear it from

8:39:42 you. Let’s give it up for you. I think you did a good job. Okay, so before we close it out, I uh first of all, I hope I see you next year, but we have one more thing from Patrick and our sponsor, Back Blaze. So, yeah, let’s hear it for Patrick. All right. >> All right, guys. The first thing I’m going to promise you is this will be exactly two minutes. Um, and let’s give it up for how much he claps. Um, because you got to clap one more time. Can you

8:40:14 do it? All right. So, uh, I’m Patrick from Back Blaze. Great to meet you all. I am buying you drinks tonight. Uh, so I am contractually obligated to explain to you what is Back Blaze. I’m going to do it as simply as I can. Um, I’ll give you one sentence to remember. AI runs on GPUs. GPUs need data. Back Blaze is the best capacity storage tier for AI data. So, if you remember anything, remember that. Two examples. uh Decart uh the

8:40:44 world model builder. Um when they were getting their start, they uh Dean called me up and he’s like, “Hey, I’ve got 15 pabytes and I’m moving it in days and I haven’t broken you guys yet. You’re awesome.” And I was like, “Awesome. Who are you? What are you doing? I have no idea.” Um but they chose Back Blaze because we don’t charge egress and we have really incredible throughput. So they were able to race different GPU clusters and create incredibly efficient AI training runs. Uh another customer, Cororeweave, we are their object storage

8:41:16 tier. They chose us because they ran a very exhaustive PC and what they found was that um from exabytes upward um our economics and our performance did not change and so they selected us because they believed we could scale with them forever. So whether you’re working with terabytes or exabytes um we are a great solution uh to ensure that your data is someplace where you control it you can use it how you want to you can use it with who you want to um and uh as an S3

8:41:46 compatible storage layer that is a quarter the cost of S3 and has free egress you’re going to pay a lot less to do it um like I said our mission which we execute for lots of genai companies and neo clouds and data scrapers and physical AI companies um is use your data however you want, wherever you want, with whoever you want. We are only going to focus on storage and we’re going to make sure every bite is there as fast as you need it. And our primary

8:42:17 goal is that you spend as little money as possible on storage because after listening to all the talks here for the last couple of days, what you guys are doing is so interesting and storage should be the last of your concerns. So definitely check us out. I will be upstairs. We have a signature uh cocktail, Lebe Blaze. Very original. Um try one, talk to me. And if you are early stage, we also have a startup program where we offer up to $100,000 in

8:42:48 storage credits, which if you can do the math, we’re a quarter of the cost of S3 is going to last uh a good amount of time depending on how fast you’re growing. So, thank you everyone. Thank you again to AI Engineer and Mistl and all of the sponsors and especially thanks to you guys. I flew here from Minnesota two nights ago. I’m flying back tomorrow morning. I’m exhausted. I have to go take care of my kids. I know how hard this is, but I think it was totally worth it. So, thank you, and we’ll see you upstairs.