continuous batching
Read the Portuguese translation →
Mentioned in
Section titled “Mentioned in”- Agents That Own Their Inference — Du’an Lightfoot & Khaja Omer, Akamai Technologies
- The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI
- How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs
- Strata vs Llama.cpp vs vLLM
Related
Section titled “Related”vLLM, speculative decoding, KV cache, llama.cpp, Grafana, FP8, decode, Kubernetes, prefill, Langfuse, Prometheus, FriendliAI