Build Hour: Prompt Caching

Quick Overview

Prompt caching provides one of the fastest ways to reduce API latency and cut costs by reusing compute when multiple requests share the same input prefix, offering discounts up to 90% on input tokens for newer models and significantly speeding up time to first token for long inputs.

Key Points: Prompt caching begins eligibility at 1024 input tokens, caching subsequent blocks of 128 tokens, and requires a contiguous prefix sent in the exact same order for a cache hit. The cost savings are substantial, with a 90% discount on input tokens for the GPT-5 family and nearly 99% off for cached audio tokens on the speech-to-speech model. Caching keeps latency roughly proportional to the generated output length rather than the total conversation length, showing up to a 67% faster time to first token for very long uncached requests. The parameter allows users to intentionally group related requests to route them to the same engine, which increased one coding customer's cache hit rate from 60% to 87%. Context engineering and cache preservation are inherently at odds; developers must weigh the intelligence boost of compaction/truncation against the cost of cache invalidation. Using the API instead of with reasoning models can increase cache hit rates from 40% to 80% because drops hidden chain-of-thought tokens, breaking the cache. Extended prompt caching allows storage for up to 24 hours by offloading the cache from ephemeral in-memory to GPU local storage, which coding customers have used to save up to 20% on input tokens.

Context: The OpenAI Build Hour session on Prompt Caching, hosted by Christine and featuring Solutions Engineer Erica and customer spotlight Siraj from Warp, detailed how developers can leverage this feature to immediately improve latency and reduce API costs without negatively impacting model intelligence. The discussion covered the foundational mechanics of caching, practical optimizations like using the prompt cache key, and real-world application in complex agentic workflows.