OpenAI's Prompt Cache Has a Secret 15 RPS Ceiling
Read the Portuguese translation →
Author: LangChain · Watch on YouTube
Video authored by the channel shown, not by IANIA; all rights belong to the creator. Transcript and summary generated automatically; the original-language content is preserved.
Original language: English
Original video: OpenAI’s Prompt Cache Has a Secret 15 RPS Ceiling · Channel: LangChain · 00:31 · Watch on YouTube
Topics: AI developer tools, Model inference and optimization
Summary
Section titled “Summary”Prompt caching lets a model reuse activations from an earlier request and add the result of a tool call, rather than processing the entire prompt again. The speaker says this can make requests through the OpenAI API 90% cheaper and can improve product speed. Cache hits are important, because missing them can undermine the product. OpenAI hashes some number of characters to route requests by default; the speaker says this works well at low volume, but high-volume use faces a prompt-cache key ceiling of about 15 requests per second.
Key points
Section titled “Key points”- Prompt caching reuses activations from a previous model run when sending back a tool-call result.
- The speaker says prompt caching is 90% cheaper through the OpenAI API than sending the whole prompt again.
- Cache hits affect both product performance and speed.
- By default, OpenAI hashes some number of characters and routes based on that.
- The default approach is described as working well at low volume.
- At high volume, the prompt-cache key is said to have a ceiling of about 15 requests per second.
- The speaker says their system is close to mathematically perfect and gets around a 95% cache hit rate.
Chapters
Section titled “Chapters”| Start | Chapter | Summary |
|---|---|---|
| 00:00 | How prompt caching works and why cache hits matter | The speaker explains how prompt caching reuses prior activations after a tool call, reducing cost and helping product speed. |
| 00:31 | Routing and the high-volume ceiling | The speaker describes OpenAI’s default hash-based routing and discusses a roughly 15 requests-per-second ceiling and their cache hit rate. |
Quotes
Section titled “Quotes”What is prompt caching? — 00:00
it’s really important that you get these cache hits — 00:00
I think it’s 15 requests per second — 00:31
we’re at like a 95% cache hit rate or something. — 00:31
Concepts and entities
Section titled “Concepts and entities”prompt caching, prompt cache key, OpenAI
Full transcript (original language)
Section titled “Full transcript (original language)”00:00 What is prompt caching? The model runs, it does a tool call, and then when I send the result of that tool call back with to request the model to run more, you want it to get a prompt cache which basically says, read the activations that you already had that you were just running for me, but just add a little bit more. And it’s, you know, 90% cheaper through the OpenAI API than it would be to to send the whole thing. Binary makes your product work or not work, basically, as well as speed it up. So, it’s really important that you get these cache hits, or otherwise you’re just you’re cooked. By default, what OpenAI does is it takes some number of
00:31 characters and hashes it and then it routes based based on that. That’s really great if you have low volume. If you have high volume, that prompt cache key, I think it’s 15 requests per second, you can be mathematically perfect which I’m not saying we are, but like we’re I we did the math and then we’re pretty close, and we’re at like a 95% cache hit rate or something.