# Prompt caching: 10x cheaper LLM tokens, but how?

Source: https://www.youtube.com/watch?v=ab2ImEEgcqU
Recap page: https://rapidrecap.app/video/ab2ImEEgcqU
Generated: 2025-12-21T17:03:32.978+00:00

---
## Quick Overview

Prompt caching dramatically reduces LLM token costs by reusing calculations from previous tokens, allowing models to save up to 80% on latency for long prompts, though it requires careful implementation to maintain accuracy and avoid non-causal prediction errors.

**Key Points:**
- Prompt caching can make LLM inference up to 10 times cheaper by reusing computations for cached tokens.
- For long prompts, caching can reduce latency by up to 80% by avoiding recalculation of the initial sequence.
- The cost saving primarily comes from avoiding the re-computation of KV (Key-Value) cache matrices for the prompt prefix.
- The reduction in cost/latency is achieved by only processing the newest token in the sequence, instead of the entire sequence every time.
- A major challenge is ensuring the reintroduced KV matrices maintain the correct context, as poorly handled caching can lead to incorrect results or hallucinations.
- The key insight is that the attention weights calculated for the prompt prefix do not change unless the prompt itself changes.

![Screenshot at 09:44: The speaker explains that prompt caching eliminates the need to recalculate the massive matrix multiplications for the initial prompt prefix, which is the core source of cost savings and latency reduction.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-09-44.jpg)

**Context:** This discussion focuses on a specific optimization technique in Large Language Models (LLMs) called prompt caching, which aims to significantly improve inference efficiency and reduce token costs. The concept hinges on reusing the intermediate calculations (specifically the Key-Value cache matrices) generated during the processing of the initial prompt context, rather than recalculating them for every subsequent token generated.

## Detailed Analysis

The video explains that prompt caching is a technique to make LLM inference substantially cheaper and faster, particularly for long prompts. The core idea is that the initial prompt sequence, once processed, generates Key-Value (KV) cache matrices that do not need to be recalculated if the prompt remains the same. By caching these matrices, the model only needs to compute the attention mechanism for the very last token being generated, leading to massive computational savings. The speaker notes that for long prompts, this can result in costs up to 10 times cheaper and latency reductions up to 80%. The process involves tokenizing the input, generating the initial KV matrices, and then only processing the newly added token in subsequent inferences, reusing the existing cache matrices. The speaker illustrates this by comparing the process to a library where index cards (the KV matrices) are reused instead of looking up every word in every book repeatedly. However, the speaker warns that implementation requires care, as the model must correctly interpret the relationship between the old and new tokens to avoid generating inaccurate or nonsensical output, suggesting that the context derived from the initial prompt must be perfectly preserved in the cache.

### Prompt Caching Benefits

- 10x cheaper LLM tokens
- up to 80% latency reduction for long prompts
- cost savings realized by caching KV matrices

### The Process

- Tokenization -> Generate initial KV matrices -> Reuse KV matrices for subsequent tokens, only calculating the newest one

### Analogy

- Reusing index cards in a library instead of reprocessing the entire text for every new word

### Implementation Challenge

- Ensuring the reused context correctly informs the next token selection, avoiding errors like repeating the first token or generating inconsistent answers.

### Cost/Performance Trade-off

- Caching provides speed and cost savings, but improper implementation leads to performance degradation (inaccuracy) rather than speed boosts.

### Mathematical Basis

- The process relies on the fact that the attention weights for the prompt prefix remain constant, allowing for the reuse of the QK^T matrix multiplication results.

![Screenshot at 00:00: Visual introduction with podcast imagery and a "Become A Member Today!" call to action overlaid on a waveform.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-00-00.jpg)
![Screenshot at 08:38: Illustration showing the concept of the massive matrix multiplication \(Q \* K^T\) that is reused during caching.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-08-38.jpg)
![Screenshot at 09:24: Visual representation of the two types of matrices \(K and V\) derived from the input prompt that are stored in the cache.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-09-24.jpg)
![Screenshot at 10:15: Diagram illustrating the reuse of the KV cache for subsequent tokens, avoiding recalculation of the prefix.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-10-15.jpg)
![Screenshot at 11:17: Speaker referencing the final step: token selection based on context and attention weights.](https://ss.rapidrecap.app/screens/ab2ImEEgcqU/00-11-17.jpg)