The infrastructure behind frontier intelligence | Runway AI Summit SF 2026

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: The infrastructure behind frontier intelligence | Runway AI Summit SF 2026 · Channel: Runway · 25:15 · Watch on YouTube

Topics: Model inference and optimization, AI infrastructure and hardware, AI models

Summary

The panel discusses how AI infrastructure is shifting from model training toward inference at growing scale, including workloads involving agents and real-world applications. The speakers emphasize that reliable, efficient systems require coordination across the entire stack, from hardware and networking to software, scheduling, and failure handling. They also describe growing physical constraints, including power availability, component shortages, and data-center construction timelines, which are driving interest in edge and more flexible infrastructure. Looking ahead, they highlight heterogeneous compute and system understandability as important needs, while anticipating much broader adoption of AI.

Key points

Inference is becoming a dominant workload as models are deployed more widely and agent-based workflows increase demand for compute.
Edge inference can help serve latency-sensitive applications and use power in locations where it is already available.
AI application developers increasingly need to understand the full infrastructure stack, including hardware behavior and cluster topology.
Ray aims to make it easier to scale Python applications across clusters while handling challenges such as work placement, data movement, and machine failures.
Failures at scale can occur at every layer, including hardware, orchestration, frameworks, and applications; some issues, such as numerical corruption, may require application-level detection.
Optimizing a single workload is not enough: teams also need to coordinate jobs and use spare compute capacity effectively.
Future workloads may be divided across different accelerator types, with each type matched to the needs of a particular stage.
Power, transformers, generators, memory, and other infrastructure components face supply constraints and long build times.
Smaller, simpler infrastructure designs and sites with existing power may help address deployment constraints.
For large-scale systems, understandability and predictability are essential because teams spend substantial time diagnosing problems across the stack.

Chapters

Start	Chapter	Summary
00:00	Introduction and infrastructure shifts	The panel introduces its focus on AI infrastructure and discusses the shift from training large models toward serving growing inference demand.
04:08	Production workloads and scaling across machines	The speakers describe multi-model workflows, rising compute demand, and the challenges Ray aims to address when scaling applications across clusters.
07:45	Failure handling across the stack	Robert explains how hardware, software, and application-level failures can appear at scale and why detecting and mitigating them requires multiple layers.
11:21	Whole-system optimization and compute utilization	The panel discusses full-stack performance, purpose-built infrastructure, and coordinating jobs so that compute capacity does not sit idle.
17:04	What future AI infrastructure needs	The speakers look ahead to heterogeneous compute, workload disaggregation, and the coordination required between different stages of AI workloads.
20:09	Physical constraints and deployment realities	The discussion turns to power and component shortages, long infrastructure lead times, and approaches that may simplify and accelerate deployment.
23:14	Limiting factors and broader adoption	The panel identifies system understandability as a key constraint and closes with the prospect of AI adoption expanding worldwide.

Quotes

inference is becoming the dominant workload. — 01:32

the models get more efficient, we put them into more things and then you spend even more than you did before — 05:11

truly everything breaks at scale. — 08:15

AI is not a database. — 22:12

we really design them with understandability and predictability as like the higher order bit. — 24:44

Concepts and entities

inference, edge inference, Ray, Anyscale, Runway, CoreWeave, AWS, Robert Nishihara, Joanna Rencinch, Cherieus Abuashi, Richard Alfons, heterogeneidade de hardware, falha silenciosa de precisão numérica, preemption, prefill, decode, reinforcement learning

Full transcript (original language)

00:00 Please welcome to the stage Anyscale co-founder, Robert Nishihara, CoreWeave SVP of Physical AI, Richard Alfons, Runway co-founder, Joanna Rencinch, and AWS Managing Director, Cherieus Abuashi. >> [music] >> Good afternoon. I’m Cherieus Abuashi. I am a Managing Director at Amazon Web Services. I lead our North America Solutions Architecture team.

00:31 Great models get the headlines. But today, we are going to look underneath them. What does it take to train, serve, and scale models? Uh I have the pleasure to moderate this panel here. I’d like to introduce the panelists for you. Joanna uh uh Rencinch. Uh she’s the co-founder of Runway, and she’s going to give us uh her perspective from an inference

01:02 perspective. Robert Nishihara is the co-founder of Anyscale, and also one of the creators of Ray. Uh he’s going to give us his perspective from an orchestration. Unfortunately, uh Richard Alfons from uh CoreWeave couldn’t be with us today. So, uh uh we hope everything is good with him. All right. So, let’s get started. Uh my first question is if you could share with us what changed

01:32 most about AI infrastructure in the past 2 years or so. Let me start with you, Joanna. >> Sure. So, AI infrastructure initially was all about training, right? Having the biggest clusters to train the biggest and best models. And of course, this is still like everybody still wants big clusters to train models. But now, inference is becoming the dominant workload. Um and also, the more widely adopted the models are, and the more we go from one person doing something to a swarm of agents um initiated by another

02:04 agent uh does something, the volume is growing exponentially. This means that we’re hitting the limits of what the infrastructure can accommodate. That’s why we’re seeing like everybody’s trying to build data centers. All the data centers are also getting canceled and getting under moratorium. So now there’s a shift towards distributing inference on the edge. And this is useful for two reasons. One, uh world models, robotics models, they really benefit for a look from low latency and edge inference. They need to be close to where they’re

02:34 actually consumed. Um and two, there’s actually a lot of power in a lot of places. So it’s sometimes the easier to kind of take the inference where the power already is and take it closer to the user than kind of build from these massive massive clusters that require a lot of resources in one place. >> Yeah, that’s pretty interesting. Thank you for your perspective. Uh Robert, how about you from your side? >> So with Ray and Anyscale, we really um look at the world through the lens of

03:05 training models, you know, deploying models, running inference, running data preparation workloads, running reinforcement learning, um these types of AI workloads. And as the scale and costs of these workloads has just grown immensely, um one of the things that’s changed the most is this the importance of vertical integration in some sense. So um you are or we’re used to thinking of a clean separation of different layers of the um infrastructure stack from

03:36 software layers to the hardware. But um increasingly, you know, the application developer who’s running building a reinforcement learning workload is going to have to reason about um hardware failures. They’re going to have to be aware of how they’re, you know, the different processes that they’re managing are mapping on to um different cluster topologies and things like that to get the most performance um and to and to build reliable applications. And so, we’ve seen um a increasing trend toward

04:08 the um you know, kind of developing these applications in a full stack way. >> Mm. Okay. Got it. So, developing the applications from a full stack is where where we’re seeing it being creative on edge in terms of power and compute is is what Dayana shared with us. That’s awesome. So, you want to I’m going to go to you uh next, which is when a model moves into a real product, right? When we’re able to put it into production and

04:39 what tends to surprise teams most about serving it in terms of speed and cost? >> Yeah, I think we’ve moved from this paradigm of one large model to there is an architecture of models, right? There is some tasks that are handled by a smaller model. Um there is a you know, again, agents that run like some very specific model types and then there is like more intelligent models that people use for coding and so on. They’ll have different context windows. So, we’re seeing um things go to a real workflow or pipeline

05:11 that actually needs to be orchestrated. All those workflows need to be distributed. But, ultimately everything just generates more demand for compute. I think there was an assumption that oh, the models will get more efficient and then we will use less compute. What is happening is the models get more efficient, we put them into more things and then you spend even more than you did before and you need even more compute. So, I think we’re now starting to grapple with the reality of like oh,

05:42 this is really not going anywhere. We need to solve this problem. >> Yeah, that’s interesting. Uh and I hear you, right? Uh, yes, uh, we can uh, do more now. So, yes, I’ll I’ll keep doing and and need more compute. All right. So, Robert, uh, what changes when that work has to scale across many machines and workloads as well? Would love to hear your perspective. >> Yeah. I mean, this actually gets at why we created Ray in the first place. >> Yeah. >> You can think of Ray, so if you haven’t

06:12 heard of Ray, Ray is a open-source uh, Python framework for basically building, you know, scalable distributed Python applications. >> Mhm. >> Right? And so, we were doing research at Berkeley, needed to scale our AI workloads, training, and so forth across clusters. And we were in the situation a lot of people were where we knew how to write Python programs on our laptop, like we knew what kind of algorithms we wanted to express. But then when you take that algorithm and you try to run the same thing across at a large scale, across a cluster, you

06:44 end up needing to solve a ton of new challenges, right? Like, the most basic is just how do I divide up the work across the different machines? How do I place the work? How do I move data efficiently from, you know, point A to point B? Um, what if one of the machines crashes? How do I How does my application deal with that? Um, all sorts of questions around resilience and efficiency and scalability. Um, and so, programming writing programs for clusters just ends up being way more complex than writing a program on your

07:14 laptop. And we started Ray to try to um, basically solve some of those challenges for you. So, if you are um, if you’re a developer, you know how to write a Python program, you can that’s all you need to know in order to run the same thing on a cluster, right? That’s not quite true. It’s it’s I’m exaggerating, but um, that’s the goal and those are the kinds of that that’s what we’re trying to achieve. >> I see. I see. Okay, great. And so, what I’d like to move to next is, like, okay, so when things break, okay?

07:45 And I’m going to stay with you, Robert, if you don’t mind. So, if you can share with the audience here a concrete example of something that breaks at scale. Uh in in we’d like to see like what is the user or developer experience, you know, as well. >> Yeah. Um that is a It’s a fascinating question. Um truly everything breaks at scale. And not just

08:15 um And this gets back to my point about vertical integration. You have to think about failure detection and failure handling at every layer of the stack, right? So, at the hardware layer, um you can have, of course, uh um you can have hardware failures. You can Sometimes that manifests itself in in some kind of CUDA, you know, exception an exception you get in your Python application. >> Yeah. >> Sometimes it could manifest itself just in um silent numerical corruption. Right? You’re running your application,

08:45 there’s no sign of anything, um but actually the data’s getting corrupted. >> Yes. >> Or could result in a straggler. Right? You have errors that can happen at the like Kubernetes layer, where um pods are not getting scheduled or taking too long at startup. You can You can have errors at the kind of like Ray or or PyTorch or vLLM layer. You can have errors in your application. Right? You can have errors in how everything comes together. >> Um >> A big challenge is um I mean, there’s error detection, like, error

09:17 localization, and just figuring out where the error was, and then, of course, the actual uh mitigation. >> Yeah. >> On the um detection side, you want to like, of course, there are passive health checks you do on every node, but one of the most interesting things we talk about vertical integration, um there’s a lot of application level health checking that people do as well. For example, I mentioned you can have silent numerical corruption um if something’s going wrong on your GPU. Sometimes in

09:48 your application, say you’re doing training, um there might be multiple times when you uh recompute the same value, right? You might have some activations in that you’re computing your forward pass. Yeah. >> That um you then throw away and then on your backwards pass you actually recompute those activations um to avoid having to store them in memory. To save GPU memory. And if you compute hashes of those activations and those vectors and then compare the hashes when you recompute them, um you can detect if

10:19 potentially if something was corrupted, right? And and so the application may actually be the one that notices um that something’s going wrong, even though they’re like, you know, all the Nvidia health checking uh tools you have didn’t didn’t catch anything, right? And so there’s different layers of the stack that are responsible for detection. There’s different layers of the stack that may be responsible for addressing the issue, right? Maybe the issue is like the first thing you should try is rebooting the machine or maybe um it’s actually uh like terminating the

10:50 application and and just use moving the compute to a different GPU, which you can do at the application level. Um Part of the what I’m trying to depict here is that um it’s really hard to solve just a a single layer of the stack. >> I see. I see. It makes sense. And actually continuing on that uh thread uh is I want to as I look at the where the layers meet, I call it, right? Whether it’s it’s on the inference or or or the orchestration or the scaling, where does optimizing, I’m going to go

11:21 to you with this uh Yuana, where does optimizing one part of the system fail because the whole system matters most, right? >> Yeah, I I think um everybody there’s so many ways to optimize the inference right now and it’s been such a pursuit that some of the best people in the industry are working on this making some incredible strides. >> Yeah. >> But ultimately there’s only so much that you can get from software. Like if you don’t understand every layer, how the software works on the hardware, how the hardware connects to each other, how like the underneath

11:52 layer of networking, how your architecture to the storage and the networking is in the first place. You will not really be able to optimize things to their highest extent and to at some point even so you’ll optimize the most that you can, you’ll just need to extend the system. I think Robert’s point at scale everything breaks. So we were running at because we serve application we survey models for applications with millions of users. We were running like billions of requests from the beginning. I think we hit every bug and managed imaginable

12:24 [clears throat] in every tool we used. So every situation we had was like, oh we never see this. It’s like, well. >> Yeah. >> [laughter] >> Um like so and even when you go below that you start to hit issues with like firmware. So we from the beginning optimize from like the BIOS to the kernel on the inference stack but then we went even lower and said, okay, basically the way that the GPU talks to the to the CPU we need the CPU to be higher performance because the performance of the CPU is

12:55 basically inhibiting the the GPU. We need the cooling system to be more efficient because the GPU keeps getting put in thermal protection. You pay a lot of money for these GPUs you want to keep them performing at the highest throughput possible. So that’s kind of how we went lower and lower ended up designing our own servers, ended up designing our own data centers and now we’re scaling our modular data centers. Which when we started to do it seemed absolutely crazy. Like nobody really thought this was a good idea for us but since then what’s happened is that again

13:26 the world of inference has grown so much that now it’s kind of the perfect thing at the perfect time for scaling edge inference. But it took a lot of conviction and belief to get there. That again, that the physical system was also part of what we needed to scale, not just the software. >> I see. I see. Okay, great. Robert, how about I’d love to hear your >> I mean, one area where we see this, you know, optimizing the the wrong thing or not keeping track

13:57 of the whole picture is when you think about there are two distinct types of problems that we think about. One is what do I need to do to optimize an individual workload, right? I’m one one developer trying to run one training job on one cluster. There’s a set of problems I have to solve to make that one training run efficient. >> Yeah. >> And then imagine now I’m a team of people trying to run hundreds or thousands of jobs across you know, many clusters.

14:27 >> Yeah. >> Right? And there’s a different set of challenges or complementary set of challenges that need to be solved there. And you can spend all the energy in the world trying to reduce cost for your, you know, your batch inference job [clears throat] or your training job or your RL run. And then, you know, maybe you really optimize your training job. But then the job finishes and the compute sitting idle, right? And so one of the most important things to set up from a cost optimization perspective is not the inside of the workload stuff.

14:59 It’s actually the cross workload prioritization and preemption and things like that. And so it’s very natural to you know, maybe you have your high priority training run that needs to get gang scheduled with a large amount of compute all at once and run and you know, have priority over everything else. Um but then you also want some sort of background elastic um inference jobs or other types of jobs that can just soak up all the unused compute. And if And

15:31 just expand to fill whatever’s not being used, so that you’re never idle. Nothing’s ever idle. So, the getting mixing those two is super important. >> I see. >> I want to add something to that. We see So, when people get AI infra from us, especially now that a lot of the top-of-the-line GPUs, there’s a wait time for them. So, you have to wait months to get them. When you’re ordering a completely new system, people tend to buy training clusters because they want to have the possibility to do both. They’re committing to this cluster for 3 to 4

16:01 years. They’re like, “Okay, I want I just you know, I can do inference on a training cluster, but I can’t do I will always do training on a pure inference cluster.” So, the way they’re thinking about it is they’re going to get a core of training and then kind of add inference on top of that. But, essentially, what it will be used for is both. Like, they’re never planned to just be like, “This cluster will just be training 24/7 for 3 to 4 years.” So, I I think it’s very important thinking of it as a system. And like, even building the tools or considering the tools of I can

16:33 even resell this capacity when it is available if I don’t use it. There’s so many solutions for that now. We’re building solutions for that now for our clients to be like, “Do you have compute?” Like, we have an on-demand product. >> Yeah, that makes a lot of sense. >> It does. It does. And and really look at the whole overall system and just don’t optimize one thing at a time. Okay, so very insightful. Um I would like to move now into what’s missing. Okay, as we look ahead, um whether it’s 1 to 3 years from now. 3

17:04 years is a lifetime, so let’s say 1 to 2 years. What capability will teams need more, you know, from that What What basically still have our today’s infrastructure handles poorly. So, as we look ahead, what do we need? What’s missing? What do we need? So, I’ll start with you, Robert, on that. >> Yeah. Well, I think there’s some trends we’re seeing already today that will be um far more

17:34 prevalent, you know, in a year or two. Um one is just the importance of hardware heterogeneity. Okay, so today it’s a little more academic where you’ll see uh people do benchmarks showing that um for example, if I split my inference into um prefill and decode stages, right? Then I might uh there are different sort of systems characteristics of each of these stages, right? My prefill part might be more compute bound and my inference part or my decode part might be more like memory

18:06 bandwidth bound. And so, I can opti- like choose different accelerator types for different stages to uh match the the performance charac- the requirements. Um I think you’re going to see far more uh disaggregation of workloads into multiple different pieces and then optimization of like usage of different um types of compute for different parts of your >> parts of that workload, okay. >> Um and you see this for for reinforcement learning, of course, because you >> I mean, it it reinforcement learning is

18:36 actually a great example because it mixes training um and inference, >> Yeah. >> right? And um you know, you’ve often run large-scale sandboxes on uh CPU machines and your inference might be split into prefill and decode. You might also have reward models, which are you know, the inference part might be more um consistent throughput, but then your reward models might require be more bursty and and want some elasticity. >> Mhm. >> Um and then like training might uh you know, has different characteristics. And

19:07 so, um all of these different things have different uh hardware requirements, different like failure handling strategies, different, you know, optimization techniques, and it’s very natural to split them out and and manage them separately. But then, of course, there’s a ton of coordination between all of these pieces, right? They’re not just running in isolation. They’re you’re constantly doing weight sync from your training to your inference, you’re moving rollouts uh back, you’re packing the rollouts into sequences, and there’s a um

19:38 a lot of uh that type of stuff that I think is just going to become more and more uh complex. >> I see. Very good. You know, hardware heterogeneity, that’s what I was thinking. Can I say that word? But that’s what it is. Different accelerators looking at the workload and what part of that workload needs maybe a different type of compute for it. Uh you want to How about the from your perspective from an inference >> So, it’s also add something that I think is very relevant, but something complementary to that is we deal also more with the physical limits. So, I

20:09 spoke a bit about um this in the beginning, but one of the issues is there’s not going to be more power plants magically in 2 years, right? These things like take 5 to 10 years to build. Um so, there’s going to have to be a change in the Actually, there’s this active change of thinking right now that we have to again go on the edge where the power is. But additionally, um we have to build the infrastructure for AI differently. For everything that you can imagine that a data center uses, there’s a massive

20:40 waitlist. That like diesel generators years, transformers years, like everything. Um components like everything, you know, for It used to be just memories. Like the memory shortage, now there’s a shortage for pretty much anything. Um everything gets bought out in advance. We don’t really feel these shortages as as much yet, though we have started to in the increase in pricing in the industry. Prices have grown like crazy for compute. And I think the reality is that no real relief coming in the immediate future because these are problems that

21:11 take years to solve like building a memory factory. Again, it’s like a 10-year thing. So, I think heterogeneous compute is one thing like distribution of other platforms and producing these in different places. >> Yeah. >> But again, looking at what can we use that is already there? You know, again, can we use sites that are like 5 megawatts that where I don’t need to upgrade the transformer? Can we operate with a system that is more simplified that doesn’t use water? Um And then we don’t need to connect it to

21:42 the water mains. We just have power and internet and that’s it. Um can we essentially build with a smaller structure so we don’t need to do 2 years of just permitting and planning applications. We kind of started with thinking about all of these restrictions and we put together you know, a solution that from the beginning doesn’t doesn’t need them. And what we’re finding is that even very large enterprise >> Yeah. >> um enterprises are very rapidly coming to grips with the fact that this is the

22:12 new reality. >> I see. >> Um and this and also AI is not a database. You know, like the things that we built for the database generation where everything has to be like, you know, tier three or tier five does not necessarily need to apply in the AI world. It’s interesting to see this shift from infrastructure being a box-ticking exercise or if you’re not compliant with every single standard, it’s a no. >> Mhm. >> To kind of people moving towards how can we get this to work? >> Yes. >> [laughter] >> How can we get this to work? And again,

22:43 we were thinking like that from the beginning. It’s a great moment, but it’s interesting to see that everything is following and the problem is real. >> I see. I see. Very cool. All right, so this is looking at say what’s missing, what do we need? I’m going to go to my last question, which is maybe in one phrase what’s going to be the limiting factor as you look ahead? Like this is what we need, but what’s the limiting factor in in in the next 3 years that you you see on your horizon? I’ll start with you,

23:14 Robert. >> Yeah, um well, you know, I mentioned the trend toward vertical integration and just optimizing everything together, and I think that’s increasingly important as scale and cost just grow. Um one of the lessons we’ve learned with Ray over time is that normally if you’re if you’re building some um software or some distributed system software like Ray, you might think about it’s obvious the importance of performance is obvious, right? The

23:44 importance of reliability is obvious. Um but one thing that we maybe didn’t fully appreciate early on and that we very much appreciate is the importance of understandability of the system. And so much so that we actually put a premium on that above everything else, you know, including performance and reliability I mean well reliability is is is essential as well. But um the reason is that for these large

24:14 workloads, um I mentioned things can go wrong at every layer of the stack, right? If you’re doing large model training or inference or anything, you are in a you are in a constant state of debugging. >> Yes. >> Right? And if you’re in a constant state of debugging and trying to figure out what went wrong at which layer of the stack, >> Yeah. >> you need to understand every single component. >> Yeah. >> And if there’s a component, a big part of that stack that is um a black box and you don’t really understand, it’s not predictable, you don’t understand how it behaves, >> Yes. >> um

24:44 at some point, uh that’s just not going to work for you, right? And that’s going to be a a limiting factor, right? And so as we design these systems, it’s we really design them with understandability and predictability as like the higher order bit. >> Got it. Thank you. You want to maybe in in one phrase if you don’t mind. >> A quick answer. I mentioned earlier one of my friends did something interesting. She said that LLMs have turned work into a slot machine. You know, like put prompt in, work jumps out and sometimes

25:15 you win and sometimes you don’t and you get to try again. So, I think this is just now being done by like us here, you know, people in the South, people in big cities, people in tech, but I think the worldwide adoption is still fully coming and I think the level at which this wave will hit is still >> [snorts] >> like about to be experienced. >> Awesome. Yuwana, Robert, thank you so much for your insights. For the audience, thank you for listening to us. Let’s give them a big round of applause. Thank you so much. >> Thank you.