Prompt Caching, Explained: When It Saves Money and When It Does Not

At 9:12 a.m. on a Monday, one of our support agents started costing 4x more than expected.

The model was not hallucinating, the traffic graph was not wrong, and nobody had accidentally switched from Haiku to Opus. The issue was simpler: every request included the same 42,000-token policy manual, but our prompt cache hit rate was effectively zero because we were prepending a fresh timestamp and user-specific metadata before the manual.

Same content. Same model. Same application. No cache savings.

Prompt caching is one of those features that sounds automatic until you try to make the bill go down. In practice, it rewards a very specific prompt shape: a large, stable prefix reused across many requests. If you understand that one sentence, you can usually predict whether caching will save money or quietly do nothing.

What Prompt Caching Actually Caches

Prompt caching is not semantic memory. The provider is not saying, “This looks similar to something you sent earlier.” It is closer to:

“The beginning of this request matches a previous token sequence closely enough, for this model and cache policy, so we can reuse internal computation for that prefix.”

That distinction matters.

Prompt caching generally helps when you repeatedly send the same long prefix:

• System instructions
• Tool schemas
• Long documentation bundles
• Product catalogs
• Codebase context
• Legal or policy manuals
• Few-shot examples
• Static retrieval results reused across many requests

It usually does not help much when every request starts differently:

• Dynamic timestamps at the top
• User-specific data before shared content
• Randomized example order
• Per-request UUIDs inside the cacheable section
• Retrieval chunks that change order or formatting
• Chat history placed before stable instructions

The core engineering move is to put stable, expensive context first and volatile context later.

Anthropic vs OpenAI: Similar Goal, Different Controls

Anthropic and OpenAI both support prompt caching, but the developer ergonomics differ.

Anthropic exposes prompt caching more explicitly. You can mark content blocks with cache controls and then inspect usage fields such as cache_creation_input_tokens and cache_read_input_tokens. This makes it easier to reason about whether you are creating cache entries or reading from them.

OpenAI prompt caching is more automatic on supported models. When a prompt prefix is reused, cached input tokens may be billed differently and surfaced in usage metadata. You usually do not place explicit cache breakpoints in the same way; instead, you structure requests so the reusable prefix is byte/token stable.

The practical difference is control versus convenience.

The implementation details vary by model and provider, and they continue to evolve. But the architectural rule is stable: caching is about repeated prefixes, not similar prompts.

The Pricing Model: Creation vs Read

The useful way to think about prompt caching is to split input tokens into three buckets:

1. Normal input tokens: billed at the usual input price.
2. Cache creation tokens: tokens used to create a cache entry, often billed at a premium or special write rate.
3. Cache read tokens: tokens read from an existing cache entry, usually billed at a discounted rate.

Anthropic exposes this distinction directly with usage fields like:

{
  "usage": {
    "input_tokens": 812,
    "cache_creation_input_tokens": 42000,
    "cache_read_input_tokens": 0,
    "output_tokens": 684
  }
}

On a later request, if the prefix hits cache, you want to see something closer to:

{
  "usage": {
    "input_tokens": 936,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 42000,
    "output_tokens": 711
  }
}

The first call creates the cache. Later calls read from it.

The economics depend on the provider’s current pricing, cache TTL, and model. Since prices change, I prefer to model this with variables in code rather than hard-code numbers into design docs.

Here is a small Python function I use when deciding whether caching is worth implementing for a route:

def cache_savings(
    cached_tokens: int,
    normal_input_per_mtok: float,
    cache_write_per_mtok: float,
    cache_read_per_mtok: float,
    requests: int,
) -> dict:
    mtok = cached_tokens / 1_000_000

    no_cache = requests * mtok * normal_input_per_mtok
    with_cache = (
        mtok * cache_write_per_mtok +
        (requests - 1) * mtok * cache_read_per_mtok
    )

    return {
        "no_cache": round(no_cache, 6),
        "with_cache": round(with_cache, 6),
        "savings": round(no_cache - with_cache, 6),
        "savings_pct": round((1 - with_cache / no_cache) * 100, 2),
    }


print(cache_savings(
    cached_tokens=42_000,
    normal_input_per_mtok=3.00,
    cache_write_per_mtok=3.75,
    cache_read_per_mtok=0.30,
    requests=100,
))

This uses illustrative prices only. The shape is what matters.

With a 42,000-token reusable prefix and 100 requests, caching is usually excellent. With the same prefix and only one request, caching can be more expensive if the write price is higher than normal input pricing.

That is the first big gotcha: prompt caching is amortization. If you do not reuse the cached prefix, you do not get the win.

The Break-Even Formula

You can estimate the break-even point with:

no_cache_cost = N * T * normal_price

cache_cost = T * write_price + (N - 1) * T * read_price

Where:

• N is number of requests using the same prefix
• T is cacheable tokens
• normal_price is normal input token price
• write_price is cache creation price
• read_price is cache read price

Caching saves money when:

T * write_price + (N - 1) * T * read_price
  <
N * T * normal_price

The T cancels out for pure cost break-even, which means reuse count and relative prices drive the decision. But token count still matters operationally: saving 80% on 500 tokens is noise; saving it on 80,000 tokens is a line item.

A useful back-of-the-envelope version:

break_even_requests =
  (write_price - read_price) / (normal_price - read_price)

If cache writes are only slightly more expensive than normal input and reads are much cheaper, the break-even point can be very low. If cache writes are expensive or your prefix rarely repeats, caching is fragile.

How To Structure Prompts For Stable Prefixes

The best prompt-caching optimization is not an API flag. It is prompt layout.

Bad layout:

Request ID: req_8f19a
Current time: 2026-02-14T09:12:33Z
User: customer_4812
Locale: en-US

SYSTEM INSTRUCTIONS:
...

POLICY MANUAL:
[42,000 tokens of stable text]

USER QUESTION:
Can I get a refund after 45 days?

This defeats prefix caching because the prompt diverges immediately.

Better layout:

SYSTEM INSTRUCTIONS:
...

POLICY MANUAL:
[42,000 tokens of stable text]

DYNAMIC REQUEST METADATA:
Request ID: req_8f19a
Current time: 2026-02-14T09:12:33Z
User: customer_4812
Locale: en-US

USER QUESTION:
Can I get a refund after 45 days?

The stable block comes first. The dynamic block comes after.

For Anthropic-style explicit caching, the shape may look like this conceptually:

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 800,
  "system": [
    {
      "type": "text",
      "text": "You are a support policy assistant..."
    },
    {
      "type": "text",
      "text": "Refund policy manual v2026-02-01:\n\n...",
      "cache_control": {
        "type": "ephemeral"
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "Can I get a refund after 45 days?"
    }
  ]
}

The exact request schema varies by API version and SDK, so treat this as shape rather than copy-paste code. The important part is that the large stable block is explicitly cacheable and unchanged between calls.

For OpenAI-style automatic caching, the prompt structure matters just as much even if you do not mark a block manually:

{
  "model": "gpt-5.5",
  "messages": [
    {
      "role": "system",
      "content": "You are a support policy assistant...\n\nRefund policy manual v2026-02-01:\n\n..."
    },
    {
      "role": "user",
      "content": "Can I get a refund after 45 days?"
    }
  ]
}

If the system message changes every request, you should not expect consistent cache reads.

A Practical Architecture Pattern

For production systems, I like to make cached prompt prefixes explicit in application code.

One simple pattern:

prompt_prefix/
  support_policy_v17.txt
  tool_schema_v4.json
  examples_v9.md

runtime_request/
  user_question
  user_profile
  retrieved_recent_orders

Then compose requests like this:

from pathlib import Path

POLICY_PREFIX = Path("prompt_prefix/support_policy_v17.txt").read_text()
TOOL_SCHEMA = Path("prompt_prefix/tool_schema_v4.json").read_text()
EXAMPLES = Path("prompt_prefix/examples_v9.md").read_text()

def build_prompt(user_question: str, user_profile: dict) -> str:
    stable_prefix = f"""
You are a support policy assistant.

TOOLS:
{TOOL_SCHEMA}

POLICY:
{POLICY_PREFIX}

EXAMPLES:
{EXAMPLES}
""".strip()

    dynamic_suffix = f"""
USER PROFILE:
{user_profile}

USER QUESTION:
{user_question}
""".strip()

    return stable_prefix + "\n\n--- DYNAMIC REQUEST ---\n\n" + dynamic_suffix

In practice, I also version the prefix. If the policy manual changes, I want that to be a deliberate cache invalidation event:

sha256sum prompt_prefix/support_policy_v17.txt

That hash goes into logs next to model name, cache creation tokens, cache read tokens, and request route. Without that observability, cache bugs are painful because the response still works; it just costs more.

A useful log line looks like:

{
  "route": "support.answer",
  "model": "claude-sonnet-4-6",
  "prefix_hash": "9d3a4a...",
  "input_tokens": 932,
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 42000,
  "output_tokens": 688,
  "latency_ms": 1840
}

Do not optimize from billing alone. Log cache behavior per route.

Latency: Often Better, But Not Magic

Prompt caching can reduce latency because the model provider can reuse computation for the cached prefix. But I would not design an SLA assuming a fixed latency reduction unless I have measured it on my own workload.

What actually happens in production is messier:

• Short prompts may show no meaningful latency change.
• Long cached prefixes often improve time-to-first-token.
• Output length still dominates many requests.
• Tool calls, network latency, and queueing can hide cache benefits.
• Cache misses create tail-latency spikes if writes are slower than reads.

A realistic benchmark setup is simple:

# warm cache
python run_eval.py --case support_refund --repeat 1

# measure cache reads
python run_eval.py --case support_refund --repeat 50 --jsonl out.jsonl

# inspect p50/p95 by cache_read_input_tokens > 0
python analyze_latency.py out.jsonl

Measure cache-hit and cache-miss requests separately. Mixing them together gives you a number that explains neither.

Gotchas I See In Real Systems

1. Dynamic Content Before Stable Content

This is the most common failure. A timestamp, request ID, feature flag dump, user locale, or experiment assignment gets placed at the top of the system prompt.

Move it down.

2. “Equivalent” JSON Is Not Equivalent

These two snippets are semantically identical but not necessarily cache-identical:

{"name":"refund_lookup","timeout":3,"strict":true}

{
  "strict": true,
  "timeout": 3,
  "name": "refund_lookup"
}

If you include tool schemas or config in the cached prefix, serialize them deterministically:

import json

def stable_json(data: dict) -> str:
    return json.dumps(data, sort_keys=True, separators=(",", ":"))

This avoids accidental cache misses from key ordering or whitespace changes.

3. Retrieval Can Destroy Cacheability

RAG systems often build prompts like:

Retrieved documents:
[doc A]
[doc B]
[doc C]

Instructions:
...

If retrieval results differ per query, the prefix differs per query. That is fine if the retrieved context is genuinely dynamic, but do not put your stable instructions after it.

Better:

Instructions:
...

Long-lived product documentation:
[stable docs]

Query-specific retrieved documents:
[doc A]
[doc B]
[doc C]

For high-volume systems, consider splitting context into:

• Globally stable instructions
• Tenant-stable documentation
• User/query-specific retrieval

Each layer has different cache behavior.

4. Cache TTLs Matter

Some cache entries expire. If your traffic is bursty, a prefix may be reused heavily for ten minutes and then go cold. Longer cache lifetimes can help, but they may have different pricing or availability depending on provider and model.

This is where request distribution matters. “We get 10,000 requests per day” is less useful than “we get 200 requests per minute for this exact prefix between 9 a.m. and 10 a.m.”

5. Model Changes Invalidate Assumptions

A prefix cached for one model should not be assumed reusable for another. If you route between Claude Opus 4.8, Sonnet 4.6, GPT-5.5, Gemini 3, or open models like Llama, Qwen, DeepSeek, MiniMax, or Kimi, track cache behavior per model.

Multi-model routing is useful, but it complicates amortization. If traffic is split five ways, each model may see fewer repeated prefixes.

6. Prompt Personalization Can Backfire

Teams often personalize system prompts:

You are helping {{customer_name}}, who is on plan {{plan_name}}.

That feels harmless, but if it appears before the cached content, every customer gets a different prefix. Put personalization later unless it truly changes the model’s interpretation of the stable block.

When Prompt Caching Saves Money

Caching is usually a strong fit when:

• The cacheable prefix is large, often thousands or tens of thousands of tokens.
• The exact same prefix is reused many times.
• The prefix remains stable across requests.
• Traffic arrives within the cache lifetime.
• You can observe cache creation and cache read tokens.
• The route has predictable prompt composition.

Concrete examples:

• A coding assistant repeatedly includes the same repository map and style guide.
• A support bot includes the same policy manual for every refund question.
• An agent uses a large static tool schema on every request.
• An internal research assistant includes the same corpus overview for a team.

For these workloads, prompt caching can be one of the highest-leverage cost optimizations because it does not require reducing context quality. You keep the long prompt; you just stop paying full input price every time.

When Prompt Caching Does Not Help

Caching is weak or negative when:

• The prompt is short.
• Prefixes are rarely reused.
• Most context is query-specific retrieval.
• Dynamic data appears before stable data.
• You constantly modify instructions during experiments.
• Traffic is spread thinly across many models, tenants, or prompt variants.
• Cache writes cost more and reads are too rare to amortize them.

A common gotcha during evaluation: prompt caching looks bad in a notebook because you run each case once. Then it looks excellent in production because thousands of users share the same stable prefix. The reverse also happens: a demo loops over the same prompt 100 times and shows great savings, but production has per-tenant customization that prevents reuse.

Test with the real request distribution.

A Simple Cache Readiness Checklist

Before enabling or relying on prompt caching, I check:

• Is there at least one stable prefix over a few thousand tokens?
• Does it appear before all dynamic content?
• Is serialization deterministic?
• Is the prefix versioned and hashed?
• Can I log cache creation and cache read tokens?
• Do I know the expected reuse count within the cache lifetime?
• Have I measured latency separately for cache hits and misses?

If the answer to three or more is “no,” caching may still work, but you are guessing.

Practical Takeaways

Prompt caching is not magic compression. It is reuse of a stable token prefix, and the economics come from paying a creation cost once and a lower read cost many times.

The engineering playbook is straightforward:

• Put large stable content first: instructions, tools, docs, examples.
• Move timestamps, user metadata, retrieval results, and request IDs after the cacheable prefix.
• Track cache_creation versus cache_read tokens per route and model.
• Use deterministic serialization for JSON tool schemas and config.
• Model break-even with your actual provider prices and traffic distribution.
• Be cautious with multi-model routing, per-tenant customization, and short prompts.
• Measure latency for hits and misses separately instead of trusting averages.

The best prompt cache optimization I have seen was not clever at all: we moved 42,000 stable tokens above 12 dynamic tokens. The model behavior did not change. The bill did.