The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI · Channel: AI Engineer · 14:28 · Watch on YouTube

Topics: AI agents, Enterprise AI applications, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

Byung-Gon Chun describes how agent workloads differ from chat: they involve multi-step tasks, tool calls, growing context, and latency that matters end to end. He argues that frontier open-weight models make production agents more economically viable, while the inference stack must also be designed around agent workflows. FriendliAI’s approach includes prefix caching, KV cache management, cache-aware routing, and agent-aware optimization. Chun presents the company’s performance claims, customers, and deployment options, and concludes that agentic inference needs to optimize task completion rather than individual requests.

Key points

The speaker says frontier open-weight models can deliver usable agent capabilities at lower token cost than closed models.
For agents, the relevant unit is a task, which can involve many model calls and tool executions; end-to-end task latency matters more than the latency of one request.
Agent workflows repeatedly append observations to context, and consecutive steps often share a large prefix.
Prefix caching reuses previously computed key-value data, reducing repeated prefill work and improving time to first token.
KV cache management includes packing active context into GPU memory, hierarchical caching across GPU, host memory, and disks, and distributed caching across replicas.
Cache-aware routing aims to send requests to locations with the needed prefix while also balancing load.
Agent-aware optimization could make scheduling, speculative prefilling, and cache eviction decisions using agent-level context.
The speaker says FriendliAI serves teams ranging from AI-native startups to global enterprises and offers model APIs, dedicated endpoints, and BYOG deployment.
Chun’s closing message is that production agent inference should optimize end-to-end task latency, reliability, and cost.

Chapters

Start	Chapter	Summary
00:13	FriendliAI and the case for open-weight models	Chun introduces FriendliAI and argues that capable open-weight models can make production agents more economical.
03:23	How agent workloads differ from chat	He explains agent loops, tool calls, growing contexts, and why end-to-end task completion is the important latency measure.
06:47	The inference cloud for agent workflows	Chun outlines FriendliAI’s four engineering pillars and describes prefix caching and KV cache management.
09:31	Cache-aware routing and agent-aware optimization	He explains how routing can preserve cache locality and how agent-level context could improve optimization decisions.
11:39	Production use, deployment options, and closing points	Chun highlights users and a customer testimonial, describes deployment options, and summarizes the case for agent-focused inference.

Quotes

That means we have to optimize for tasks, not just individual requests. — 03:23

Consecutive agent steps usually share a huge prefix. — 05:05

Since Asian steps share a large prefix, we compute key value for that prefix once and cache it. — 07:54

So the goal is to reduce ant latency not just make one call look fast. — 10:35

Agents are not just chat with more cores. — 13:54

Concepts and entities

FriendliAI, Byung-Gon (Gon) Chun, agentic inference, continuous batching, Orca, vLLM, GLM-5.2, Opus 4.8, KV cache, Kilo Code, MiniMax, Kimi, LG, BYOG

Full transcript (original language)

00:13 Let’s uh get started. Uh hi everyone. Thank you for coming. Uh this is the late afternoon in the last day. Um so I really appre appreciate it. I’m gone, founder and CEO of friendly AI. Today I want to talk about agentic inference. So I’ll first walk through what changed uh why it matters and how we rebuilt the inference cloud for agents. Before we go deeper let me briefly

00:43 introduce friendly AI. Friendly AI is the frontier AI inference cloud for agents. So we run inference for agents at scale faster, cheaper and more reliably. We are born from a research team at S National University and those research roots still define us. We are the team that invented continuous batching. The inference optimization that is now standard across the industry and our orca work inspired 3LM a widely used open source framework.

01:16 Today we operate globally headquartered in San Francisco with a team in soul to scale frontier inference. As you know 2026 is a year agents go into massive production and it’s driven by two trends coming together. First agents are going exponential. AI agents are driving explosive adoption across software operations and knowledge work. Second, open rate motors have reached the frontier and make agents economic.

01:51 They now rival closed frontier motors in capability, which means you can run frontier quality agents on open motors with much lower token cost. Let me make the openweight motor part concrete. Open weight motors are now strong enough for these types of real agentic workflows. Here we gave the exact same task building a tower defense game with a coding agent to two models. On the left is GLM 5.2 an openw rate motor

02:22 running on friendly AI. On the right is anthropics opus 4.8. The important point is not that the outputs are identical. The point is that both complete the task at a level that is clearly usable for many agentic workflows. Open weight models have crossed the quality threshold but the economics are very different for the same task. Opus 4.8 cost about a

02:53 $150 GLM 5.2 unfriendly AI cost 27 cents about 5.6 times cheaper. So this is the promise I mentioned earlier. Open rate models give you frontier quality agents at a fraction of cost. But motor cost is only one part of the story. To make agent actually fast and reliable, the inference stack itself has to change. So let’s look at what

03:23 actually happens inside an agentic workload. So first let’s look at changes in the workload. In the past the dominant usage was chat. The basic unit was a request. A person asks a question, the model answers and the person reads it. Latency meant how fast did I get one response. Agents are different. The basic unit is a task. A task may involve many model cores, many tool cores and it may run autonomously for a while.

03:58 So the user does not really care about the latency of one individual request. The user cares about when the whole task is completed. That means we have to optimize for tasks, not just individual requests. Let’s look at agent workflows more closely. An agent really runs a session made up of tasks. Each task typically runs in a loop. First, it plans, which usually means an LLM call. Then, it acts maybe by calling a tool. Then it

04:33 observes the result and adds the that back into the context and it repeats this until the task is done. So we are constantly alternating between LLM inference and one or more nonLM tool executions. So there is a gap between LLM calls. An agent can also create sub agents and run them in parallel. Agent inputs also look very different from chat. The graph here shows the

05:05 prompt and completion length distributions of our internal coding agent runs with GLM 5.2 which we use day-to-day. They are much longer. They grow as the task progresses since every observation gets appended back into the context. There’s an important pattern here. Consecutive agent steps usually share a huge prefix. If we recomp compute the same prefix every time, we are burning a lot of compute on work we already did.

05:40 So this is one of the biggest opportunities in agentic inference. So how token hungry are agents? Now let’s look at a long horizon task example like deep research. We ran explained the spec decoding framework in VLM using code with GLM 5.2 unfriendly AI. There are multiple stages and each stage is composed of sub agents which run multiple inferences and tool cores. So it might run tens or even hundreds of

06:16 inference steps sometimes over minutes or hours and the shared context keeps going the whole time. For the user what matters is not the latency of a single token or one core. What matters is when is my test completed. So agent inference is not just chat with more requests. It’s a different problem. The context grows over time. Tour work is interled between model cores. The

06:47 number of model cores depends on the input. So you can’t really plan around a fixed request rate plan. And the real metric is end to end test latency, not a single request latency. This is where friendly AI comes in. We rebuilt the frontier inference cloud specifically for agentic goal close around the challenges I just walked through and we set one goal optimize end to end test latency the task not just the request. So how do

07:21 we do that? Let me show you the key engineering behind it. Here’s the engineering map for how we think about it. We built the stack layer by layer around agent workflows. There are four big pillars I’m going to cover today. Prefix caching key value in short KB cache management, cash aware routing, agent aware optimization and of course underneath we need model layer optimization like sparse attention

07:54 for long context techniques to reduce errors, fast corners, resilience serving and more. In this talk, I’m going to focus on the four pillars. Let’s start with prefix caching. Since Asian steps share a large prefix, we compute key value for that prefix once and cache it. Then on later steps, we reuse the cache key value and only process the new suffix. Reading from cache is much cheaper than recmp computing prefill. So this

08:28 improves time to first token and reduces compute on every step. And the longer the task runs in agents, the more valuable this becomes. But caching only works if the KV cache actually fits and can move around efficiently. So we need strong KV cache management. We use frugal memory management to pack more active context onto each GPU memory. We use KB

09:01 contigation to reduce the memory footprint. We you we use hierarchical caching across GPU memory, host memory and disks. So we can go beyond GPU limits. And we also use distributed caching. So one prefix can be served across replicas, not just inside one instance. At global cluster scale, routing becomes really important. A knife load balancer

09:31 may spread requests evenly across GPU clusters, but it can destroy cache locality. A cache aware router at a global scale does something smarter. It sends a request to part that already has the right prefix cached turning a cord prefill into one cache ship. At the same time, it still has to balance load. So one part doesn’t become a hot spot. In this example, the two requests of task A go to the same part one for cache

10:04 locality. The next piece is agent aware optimization. And this is the next fronture of agentic inference. Today, most systems schedule each LLM core as if it were independent. They don’t really understand that this core is part of a longer agent program. But if the optimizer knows the agent level context, we can make better decisions. For example, preempting the

10:35 right work, speculatively prefilling context for an unlikely next step or making a better cache eviction decision based on agent level context. So the goal is to reduce ant latency not just make one call look fast. When we put all of this together, this is the payoff. We are using the same model GLM 5.2 with kilo code to create a simple mobile game. We ran the same task

11:09 with model APIs of friendly AI and another well-known inference provider. As you can see, friendly completes the same task end to end tox thanks to our Asenticentric cloud design. So what does this unlock in practice? a stronger production agent stack. Take an agent you already like. Now plug in openweight frontier models like GLM

11:39 5.2, Minimax and Kimi served on friendly AI. The motor gives you frontier quality capability and better economics. Friendly AI gives you the speed, reliability and endtoend test performance needed in production. That combination, quality, speed, reliability, and cost is what makes agents actually useful and economical in production. Friendly AI is currently powering teams in production from AI native startups to

12:14 global enterprises. I’d like to highlight a couple here. Hilo is a hugely popular Asantic AI coding tool serving millions of users. LG is a global enterprise whose businesses range from electronics to healthcare to energy. Very different companies, but they all need the same thing. Fast, reliable, cost effective agentic inference.

12:45 This testimonial from our client Kilo says it all. Over the past year, Kilo Code has tested several inference providers hosting both open and closed models. In a split test of GLM5 usage compared against other thirdparty providers and direct usage from the model lab G.A.I., Friendly AI was consistently seven times faster with a significantly lower error rate. Today, friendly AI is a core component of the killer stack.

13:20 And you can consume this however fits your stack. Model API is the fastest way to start. Core frontier open weight models through our serless API. Dedicated endpoints give you your own isolated deployment with guaranteed SLAs’s for production workloads. And BYOG bring your own GPU lets you run friendly inference on your own infrastructure. Same stack, three ways to deploy. To wrap up, there are three things to remember. First, frontier open weight

13:54 models make production agents economically scalable. Second, agents are not just chat with more cores. Agentic inference requires optimizing end to end task latency with the challenges I mentioned. Third, friendly AI is built as an inference cloud for that word. Fast, reliable, cost effective agentic inference. Thank you for attending my session. If you’re building agents, give uh frontier

14:28 openweight motors a try on friendly today. You can get started at friendly.ai in minutes. And uh thank you. I’ll be around after the session. Um thank you. >> [applause] [music]