Question:
Imagine that your LLM API cost suddenly doubled in the last month. The deeper analysis reveals that, while the user inputs appear different from a text perspective, most of them have semantically similar meanings. How would you, as an engineer reduce the redundancy in this input without affecting response quality?
What Is Prompt Caching?
The AI system uses prompt caching to reduce costs and improve performance. The system uses previously processed prompts such as static instruction, prefixes or context to reuse content. The system saves both input and out put tokens.
Imagine a planning assistant for travel where the users ask frequently questions such as “Create a 5-day itinerary for Paris focused on museums and food.” Even though different users may phrase their requests slightly differently, they will still have the same core purpose and request structure. If the model is not optimized, it will have to repeat the computation every time and increase both cost and latency.
With prompt caching, once the assistant processes this request the first time, the repeated parts of the prompt—such as the itinerary structure, constraints, and common instructions—are stored. The system will reuse the content it has already processed when a request similar to the first is made. The system can respond faster and at lower API cost, and still deliver accurate and consistent results.
What gets cached, and where is it stored?
At a high level, caching in LLM systems can happen at different layers—ranging from simple token-level reuse to more advanced reuse of internal model states. In practice, modern LLMs mainly rely on Key–Value (KV) caching, where the model stores intermediate attention states in GPU memory (VRAM) so it doesn’t have to recompute them again.
Imagine a coding assistance with a system instruction that is fixed. “You are an expert Python code reviewer.” Every request contains this instruction. Once the model has processed the data, it stores the attention relations (keys and value) among its tokens. Models can reuse the stored KV state and calculate attention only for new inputs, like the code snippet.
Prefix caching extends this idea across all requests. If multiple prompts start with the exact same prefix—same text, formatting, and spacing—the model can skip recomputing that entire prefix and resume from the cached point. It is particularly useful in pipelines for chatbots, RAGs, and agents where long instructions and system prompts are rarely changed. This results in reduced latency, and reduced computation cost. The model can still fully understand new context and react to it.


Structure Prompts to Get High Cache Rates
- Move user-specific content or that which changes to the end of the prompt.
- Even small modifications can reduce the reuse of prefixes.
- To avoid cache misses, serialize structured data in an order and format that is consistent.
- Monitor cache hits and group requests that are similar to maximise efficiency.
You can also read our conclusion.
It is important to maintain response quality while decreasing the amount of computation. A good approach would be to examine incoming requests for common structure, prefixes or intents and then to restructure the prompts to ensure that all calls are consistent in terms of reusable context. The system can avoid processing the same data repeatedly. This reduces latency, and API costs.
For applications with long and repetitive prompts, prefix-based reuse can deliver significant savings, but it also introduces practical constraints—KV caches consume GPU memory, which is finite. Memory tiering or cache eviction become more important as the usage grows. This is to ensure that performance improvements are balanced with limited resources.


