# Build Hour: Prompt Caching Source: https://www.youtube.com/watch?v=tECAkJAI_Vk Recap page: https://rapidrecap.app/video/tECAkJAI_Vk Generated: 2026-02-18T22:02:22.799+00:00 --- ## Quick Overview Prompt caching provides one of the fastest ways to reduce API latency and cut costs by reusing compute when multiple requests share the same input prefix, offering discounts up to 90% on input tokens for newer models and significantly speeding up time to first token for long inputs. **Key Points:** - Prompt caching begins eligibility at 1024 input tokens, caching subsequent blocks of 128 tokens, and requires a contiguous prefix sent in the exact same order for a cache hit. - The cost savings are substantial, with a 90% discount on input tokens for the GPT-5 family and nearly 99% off for cached audio tokens on the speech-to-speech model. - Caching keeps latency roughly proportional to the generated output length rather than the total conversation length, showing up to a 67% faster time to first token for very long uncached requests. - The `prompt_cache_key` parameter allows users to intentionally group related requests to route them to the same engine, which increased one coding customer's cache hit rate from 60% to 87%. - Context engineering and cache preservation are inherently at odds; developers must weigh the intelligence boost of compaction/truncation against the cost of cache invalidation. - Using the `responses` API instead of `chat/completions` with reasoning models can increase cache hit rates from 40% to 80% because `chat/completions` drops hidden chain-of-thought tokens, breaking the cache. - Extended prompt caching allows storage for up to 24 hours by offloading the cache from ephemeral in-memory to GPU local storage, which coding customers have used to save up to 20% on input tokens. **Context:** The OpenAI Build Hour session on Prompt Caching, hosted by Christine and featuring Solutions Engineer Erica and customer spotlight Siraj from Warp, detailed how developers can leverage this feature to immediately improve latency and reduce API costs without negatively impacting model intelligence. The discussion covered the foundational mechanics of caching, practical optimizations like using the prompt cache key, and real-world application in complex agentic workflows. ## Detailed Analysis Prompt caching functions as compute reuse by skipping processing for input prefixes already seen, starting eligibility at 1024 tokens and caching in 128-token blocks, provided the input order is identical. This mechanism yields significant financial benefits, offering up to a 90% discount on input tokens for GPT-5 models, and latency improvements, making time-to-first-token proportional to output length rather than total input length. The cache stores key-value tensors, not raw tokens, and is ephemeral by default (5-10 minutes) but can be extended to 24 hours using `prompt_cache_retention: 24_hours`. Optimization relies heavily on maximizing cache hit rates; the `prompt_cache_key` is crucial for routing similar requests to the same engine, as hashing only the first 256 tokens can lead to fragmentation across different machines under high load, a scenario where Warp saw its hit rate more than double. Context engineering, such as truncation or summarization to manage context windows, conflicts with caching because any change invalidates the cache; developers must evaluate if the intelligence gain from compaction outweighs the cost of cache invalidation. Specific API choices matter, as using the `responses` API instead of `chat/completions` preserves reasoning tokens, boosting cache hits from 40% to 80% while also improving intelligence. Warp utilizes caching across global, user, and task scopes, noting that static components like system prompts and tools should be kept consistent to maximize reuse. ### Prompt Caching Fundamentals - Caching starts at 1024 tokens, caching in 128-token blocks - Cache hits require an exact, contiguous prefix - Caching is implicit, automatic, and supports multimodal inputs (audio, text, images) - Default cache is ephemeral (5-10 minutes) but can be extended to 24 hours. ### Impact of Caching - Latency reduction keeps time-to-first-token proportional to output length, not total input length - Pricing discounts are 50% for GPT-4, 75% for GPT-4-Turbo, and 90% for GPT-5 models - Audio token caching receives almost 99% discount on cached tokens. ### Optimization Strategy 1 - Prompt Cache Key: The key hashes the prefix plus the key, grouping related requests to the same engine, which increased one customer's hit rate by 27% - The key helps manage branching chat flows where initial prefixes might otherwise get mashed together due to load balancing. ### Optimization Strategy 2 - Context Engineering Balance: Context engineering (shaping context) and cache preservation are at odds - Compaction strategies like truncation or summarization must be weighed against the intelligence boost they provide versus the cost of invalidating the cache. ### Optimization Strategy 3 - API and Tool Usage: Use the `responses` API over `chat/completions` when using reasoning models to preserve chain-of-thought tokens and increase cache hits from 40% to 80% - Use the `allowed_tools` parameter to specify curated tools without invalidating the cache, as tool definition changes break the cache. ### Warp's Real-World Application - Warp uses caching across global (static system prompts/tools), user (stable configuration across parallel agents), and task scopes (turn-by-turn agent loops) - Warp found that task-scoped prompt cache keys significantly improved their cache hit rate, passing savings to users. ### Cache Miss Checklist - Common misses occur if the prompt is under 1024 tokens, if too much time passes (without extended cache), if input content mismatches, or if using Batch API with pre-GPT-5 models - In-memory cache is ZDR eligible, but extended caching (stored on GPU local storage) is not.