Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod

Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.

Original language: English

Original video: Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod · Channel: AI Engineer · 16:00 · Watch on YouTube

Topics: AI evaluation and benchmarks, Model inference and optimization, AI infrastructure and hardware, Local and open-source models

Summary

Chris Bella, a forward deploy engineer at RunPod, argues that teams should benchmark VLM deployments rather than rely on default recipes or guesswork. He presents BenchPod, a planned template that centralizes vLLM configuration, benchmarking, metrics, and recovery features to make testing and production workflows easier. A demo compares batch-token configurations under different concurrency levels and shows how tuning can reduce inter-token latency. He also emphasizes choosing models and hardware based on the task, workload, and performance metrics needed, rather than defaulting to the newest or largest model.

Key points

Choose a model and context window based on the intended task, such as coding agents or summarization, and the expected number of users.
Default vLLM recipes are a useful starting point, but should be tested and tuned before production.
Version compatibility matters because model support and vLLM code change quickly.
BenchPod is presented as a planned template for configuring models, running benchmarks, and viewing deployment metrics in one place.
The demo includes live metrics, stored benchmark data, configurable experiments, and a watchdog that can self-heal.
Benchmarking at different concurrency levels helps reveal how configuration changes affect performance.
In the demonstrated experiment, increasing the maximum batch tokens did not notably change time to first token, but reduced inter-token latency.
Optimize for the metrics that matter to the use case, such as ITL, TTFT, or TPOT, then test with the agent before scaling out.
A frontier model is not always necessary; model and GPU choices should reflect workload needs and cost.

Chapters

Start	Chapter	Summary
00:01	Why benchmark VLM deployments	Chris introduces the challenge of moving from rapidly changing model releases and default configurations to data-backed production decisions.
03:17	Introducing BenchPod	He describes BenchPod as a planned vLLM-based template intended to simplify benchmarking, configuration changes, and production workflows.
06:02	Dashboard and deployment demo	The demo covers model and GPU considerations, dashboard metrics, deployment, and monitoring a running Kimi instance.
09:43	Comparing benchmark configurations	Chris compares runs at varying concurrency levels and explains how a higher batch-token setting affected inter-token latency.
11:17	From benchmark results to production	He shows an agent using the deployment and explains how benchmark data can inform scaling to clusters or serverless.
13:22	Tuning, model choice, and next steps	Chris recommends testing configurations against relevant metrics and choosing models and hardware according to the workload rather than frontier status alone.

Quotes

how do you actually have data? — 01:03

I created this thing called it’s called BenchPod. — 03:47

the inner token latency actually decreases quite a bit when you increase that batch tokens. — 10:46

Don’t just start with like the VLN recipe. — 13:53

just because it’s the the frontier model, maybe it doesn’t make sense. — 14:55

Concepts and entities

Chris Vela, RunPod, vLLM, BenchPod, Kimi K3, CUDA, B300, FastAPI

Full transcript (original language)

00:01 [music] >> My name is Chris Bella. I work at RunPod. I’m a forward deploy engineer. Today I’m going to be talking about benchmarking VLMs. Kind of like the struggle that people have when like especially like a new model comes out. Just some like learnings that we’ve had like internally at RunPod and with like customers. So today I’m going to be showing you

00:32 essentially like a dashboard that is built on top of like VLM. It’s using our like pods actually. So to give you like a little more context of like what RunPod does is we’re in the new cloud. We make it really really easy for you to rent GPUs and starting from like R&D to like more service production workflows. A lot of times what I see with customers especially with VLM is like someone wants you know the latest model. They don’t know how to like spin it up or they they want to start going to production and they don’t know the the knobs to like touch. They’re they’re

01:03 just using whatever like the base config and I’m guessing a lot of you probably seen like VLM recipes before. You know it’s a good start especially but like especially in this world we’re always moving fast. So how do you get to more production level style of code and at the same time not just like vibing it. It’s more like how do you actually have data? How do you show you know your manager like your staff engineer that hey this is actually better. This is actually working correctly. So

01:35 really what I’m trying to say is like I’ve seen a lot everyone’s just guessing. Like you know you’re a lot of times I see people are using the wrong VLM version. They might they’re probably not using like the nightly version. They have the wrong size for their context window. It really depends like I I think it always is a step back. It’s like what are you trying to achieve with your your LM model? Are you is it going to be a coding agent? Is it more kind of like you’re just um you know, doing summaries, like very very basic tasks. That’s usually the first question I’d be asking before you you know, getting Kim K 3, having you

02:09 know, a million long context. Like that’s not always necessary. Uh and at the same time it’s like how many users are going to be using this? Um a lot of times people are like going everyone’s trying to always get like the the smallest amount of GPU for the cheapest price. But if you’re going to be going to a more concurrent users, you need to start thinking about like how am I going to be utilizing the KB cache? Uh so all this is there’s all these little like optimization knobs that you need to like figure out before you go to production. And a lot of times we’re trying to move fast. So um

02:43 Yeah, like like I said, like RunPod like we like have very much like, you know, we make it very very easy. It’s like a VM. Uh it’s dockerized. Uh you can bring your own image. But at the end of the day like um like who here has used like v l m >> before? >> Okay, so you know, like when you’re looking for like like especially like Kim K 3, like if you you use like a version of like, you know, 0.27, it’s not going to be working. Like like we’re moving so fast the code is not getting as fast. Like you need to have the right version. And a lot of times that turns into kind of like a CD

03:17 CI CD process that um especially if people who don’t understand what’s happening uh just start struggling. They they start seeing errors and they’re like, “Why is this not working?” You know, it the the layers within it are not they haven’t been written or they’re it’s very in development code. Uh and at the same time that’s actually kind of a problem is like Kim K 3 comes out, you know, the next day I would not put that on production just yet. Uh you’re going to be seeing a lot of probably hallucinations. Uh it’s just not ready in that state, I would say.

03:47 Um so what I propose um so uh I created this thing called it’s called BenchPod. It’s going to be a template. It hasn’t come out yet. Um so it’s essentially built on top of like uh v l m. Um it’s a whole metric system. Uh you can benchmark very easy. And a lot of you who are familiar with like v l m you can benchmark already, but this kind of um centralizes everything, so you can be able to see everything, be able to change the config very quickly. Uh that’s like a very important thing that I I notice is that uh especially with people who use like our uh our pods is

04:22 you set up a config, but then you have to restart the image. You have to wait for cold starts to happen. It’s all this loading, and you’re not moving as fast. You’re waiting like 30 minutes like for Kim A I 3 to like download the weights, load into GPU. Uh it’s kind of wasted time. I mean, if you have an agent, you know, you really don’t care, but um that is a pain point especially when we’re trying to move fast. And at the same time it’s like the cool thing about Runpod is you can find these machines. So, if you have that question, it’s like I want to run Kim A I 3 on a B300, we have B300s, and you can start going from

04:55 like not just like reading about it. You can actually start testing it and seeing if, you know, there’s truth to all these metrics. So, yeah, so like bench like like I said, like bench pod is more like um it’s it’s built like a of a a docker image of um um v l m open a b i. We’re going to be having every version already. Um we just have a little like like payload that like pretty much creates this whole fast a p i application. Uh you can use it for your agents. Uh the cool thing is you can serve uh you can pretty much specify which uh models you

05:29 want to have. We have a whole watchdog, too, so it like self-heals, too. So, uh if you were actually starting to go to more production levels, uh you can actually uh you don’t have to worry about it. And then like I will show you uh like this past Sunday uh Kim A I went down on our instance, and it uh self-healed. And, you know, we have we have right now like four nodes running it, so none of like our internal users saw any problem. It was a production ready um didn’t have to like think about it too much. And at the end of day, yeah, like it’s it it makes it much more sim- uh simple

06:02 process to uh have production work workflows. So, I’m guessing a lot of people I’m I’m going to show you Kim and K3 because we’ve been using that a lot. Um I’m guessing a lot of people here are very familiar with Kim and K3. Um so, you know, 1 1 million contacts. Uh one thing that’s interesting on 1B 300, uh can’t get to the 1 million. You got to use half of it. But one one thing I’ve noticed especially with our team, like we’re using it a lot for coding uh agents. Um that is very much um

06:33 enough to do long task. Um It’s utilizing KB cache uh very well, too. So, um in terms of speed of like time to first second, um it is very very very quick. And like in the demo, I’ll show you too with like my coding agent running this Kim and uh model on RunPod. So, this is like our dashboard. Um It has a lot of metrics. You know, it can it look looks like a lot, but uh essentially, you know, we’re always thinking about like the GPU, like, you know, have a lot of like GPU uh metrics, you know, and what CUDA version you’re

07:05 on, which is very very important. You know, if you’re running Kim and it doesn’t work on a 12 8 CUDA, it works on 13.0. Uh and really it’s what we have is like the best we can we we can look at benchmarking, look at all our data, we can like playground with it. So, like before you even put it into your like coding agent or uh just hitting the OpenAI uh response, you can start testing real quick if this is working. And just a lot of like more CICD process of like, you know, how long are my cold starts happening? Um how are how are my metrics looking um

07:37 through actual traffic? So, I’m going to show you guys the demo. So, So, really like I’m going to show you real quick like RunPod itself. So, um this will So, we don’t have this set up yet. Like I said, um it’ll be up probably at the end of the month. Uh but really all we have is just like a Docker image. We can be able to uh put that in right there. If it loads correctly. But uh Uh, yeah, so right now there’s no B300s, but uh, I have one running right now.

08:09 That’s the fun thing about working in this industry. Uh, you know, uh, supply can be uh, dependent. Uh, but >> [laughter] >> But yeah, it’s in in the sense of that like, you know, we make it really easy. You can put storage volume. Uh, you deploy it, you know, we have a part two so you can deploy when you’re available. So, you know, if you were seeing the screen like this, it will deploy whenever it get it get started. Uh, so that’s like literally the process. You click that button, benchpod like starts up.

08:40 Um, and I can show you like right here. I have it on the wrong one, but I’ll show you guys here. So, like this is like this is our actual like Kimi running currently. Um, so a cool thing is like you know, you you start like it has a whole air scan so you have actual data to know like what is happening. Um, cool thing is we have live metrics, too. As like things are running. Uh, all of this is also like backed up in like storage data. So, like it’s all

09:11 um, um, backed up no matter what. Um, it’s all face fast API calls so your agent could easily just call all this data. You can actually just serve everything. Like you don’t even have to use the UI. I I think I’m a big proponent of like as we go to agents it’s like I still want to have like some awareness of what’s happening and having this gives me a good like like eagle’s eye view of like what is happening with my uh, Kimi K3 K3 instance. Uh, but yeah, like you can see here like I was going to show you like uh, this past weekend we had a failure with a GPU. Uh, we were able to fix it,

09:43 but uh, being able to just do a cold start time like 9 minutes on it like K3s pretty pretty good. Um, and the cool thing here is that we have the serve so you can start out experiments and essentially like apply them. It’ll restart real quick. Um, you can it’s literally just like the config YAML file from like VLM. And um, so like for this like little demo what I tested was we were just changing the max number of batch tokens uh, that were sent through. And I kind of wanted to show you guys of

10:14 like, you know, what what kind of like speed ups would happen with this. Um I’ll go right here. So, I run these before because we’re not going to spend like 20 minutes just looking at benchmarks. So, these benchmarks are running like concurrency. So, they’re they’re running um you know, starting with like just one concurrent user and then eight, 64, 192. And the cool thing is like we can be at we can actually compare multiple runs against each other.

10:46 You can kind of see like even just increasing the match the the number of batch tokens doesn’t really change that much the like time to first second. But, interesting thing that I did notice of increasing that is like the inner token latency actually decreases quite a bit when you increase that batch tokens. Which means it’s like the request itself, if it’s especially running like a long like job essentially, it’s actually much faster finishing the

11:17 job than the base like config that like you find on VLM recipe. The cool thing is is like you can add this to an agent and just start going crazy with it. And you can start getting those answers much much quicker, have the data, and move from this if you have it like on a pod, you can start moving to, you know, a multi-node cluster. You can move to serverless, too. So, if you’re going to be like scaling up and down that that this is a perfect way of like

11:48 actually getting data. And just to give you like a context on our serverless, like serverless a lot of people will do the same thing. They they just put whatever the base config is and then are struggling as like they’re waiting for workers to start up. This actually gets you the real data. You’re now ready to go to production. Makes it really really simple. And just gives It’s like I always say, it’s basing everything in facts. And I can like show you guys right here. Like this is actually like Kenny writing. So I was just like, what is a

12:19 bench pod? I misspelled it. Sorry. But that this is little command instance running through it. Um yeah, and Um you know, it is pretty pretty quick. Like like I would say like there’s like a probably like 50 users right now on our internal one and we most of us are getting off cloud and just using the and we have other instances with like GLM, uh deep seek, uh quant 3.8. Um but yeah, like I think in the future we’re looking to more at like smart routers. Like how how do we

12:52 host all this without being reliant on someone else? Um Yeah, so let me go back to our uh thing real quick. So some like important data like we were just talking about it. If anyone wants to screenshot that. But uh Like I said, like you can you can specify the experiment yourself too. So if you were like, I’m going to have like 500 people using this all the time. Like I want to like see how well this is going to be. Um

13:22 But at the end of the day, like I said, it is how do we go from R&D to production? And I think this is where like bench pod will make it a lot more easier for people to use. Um So like at the end of the day, like what I want you guys to like know is like um you know, pick a pod, get like a GPU machine. You have your model, compare configs. Don’t just start with like the VLN recipe. The VLN recipe is proven, but it’s not production ready. Um and then

13:53 test. Have your agent tested. And then whatever, you know, based on what you need, if it’s ITL, if you need TTF, TPOT, like metric, like optimize on that. And then ship that winner. Go to service, scale out, and now you can like um Now you have a working LLM model that can be used internally, can be used for our Um Yeah, it’s it just makes it very very simple. But yeah, one thing I would say especially with

14:24 like Kimmy that I’ve noticed is um it’s not always necessary for a team. Like like a big thing that I think like most people don’t think about is that um you can use a smaller model. You know, it it’s a B 200 right now like a node. That’s like on our on our platform like that’s going to cost you 40K a month essentially. And it might go up, it might go down a little, but you know, it’s a very powerful machine and I think

14:55 when you’re thinking about like how you’re going to be um serving stuff if you want to be doing this internally or for yourselves is like what makes sense the best. And just because it’s the the frontier model, maybe it doesn’t make sense. Um like I said like like I run a deep six on a 4 H 200s and does a really good job. Kimmy’s more for like long tasks that I’m trying to solve something like very very detailed. Um but like I said, the other thing is like making sure your like tooling

15:26 choice is like on there. That’s a big thing. Context like I I’m not going to be going through too much about like context like wikis and all that, but the ability to have all that and like AI governance is the next step that we are going to be going with RunPod, but just something to be always be thinking about especially if you’re going to be self-hosting your um your LLM models. But yeah, is there any questions? Like And we’ll also be at the booths. So like downstairs. But yeah, add me on LinkedIn if you want to talk more. I’m always

16:00 exploring every new model like every week. So always have some like gotchas that I’ve been learning through the through the time I’ve been doing all this work. And yeah, also scan the card QR code so you can get you get some credits and you can test out RunPod. And and said, the template will be out like at the end of the month. So you can go play around with it.