Prompt Caching and Context Windows in 2026: Cut Costs

THE AI EDIT12 MIN READ

An AI feature rarely becomes expensive because one prompt is enormous. Costs balloon when every request resends the same operating manual, support history, and tool definitions. In 2026, cached input on OpenAI’s GPT-5 mini is $0.025 per million tokens versus $0.25 uncached—a 90% discount, per OpenAI’s published pricing. That price gap can matter more than trimming 200 tokens from a prompt. The practical goal is not to make context as small as possible; it is to send only useful context, keep stable instructions reusable, and reserve expensive generation for answers that need it.

Why This Matters in 2026

Long-context access is no longer the main constraint. How effectively teams use that capacity determines whether it saves money or creates a new cost center. For 2026 budgeting, these published vendor list prices provide a useful baseline; verify them against the provider’s current pricing before committing to a budget.

Long-context models now routinely process hundreds of thousands or millions of tokens. GPT-5 provides a 400,000-token context window, Claude Sonnet 4.5 offers a 200,000-token context window, and Gemini 2.5 Pro accepts 1,048,576 input tokens. A large context window serves as capacity rather than a recommendation to send everything available.

Cached input is materially cheaper than uncached input. GPT-5 mini lists cached input at $0.025 per million tokens versus $0.25 for fresh input, a 90% discount. Claude Sonnet 4.5 charges $0.30 for a five-minute cache read versus $3 for standard input, also a 90% discount. Gemini 2.5 Flash lists cached input at $0.03 per million tokens versus $0.30 for fresh input, a 90% discount, plus $1.00 per million tokens per hour of storage. Anthropic has also deprecated Claude Sonnet 4.5, which retires on November 30, 2026; Claude Sonnet 5.5 ($2 input, $10 output, 1M context) is the recommended replacement, so treat the Sonnet 4.5 figures above as a legacy baseline rather than a planning target.

Generation remains the premium part of a request. GPT-5 mini output costs $2 per million tokens, Claude Sonnet 4.5 output costs $15, and Gemini 2.5 Pro output costs $10 at the standard tier. Cutting unnecessary output and retries can therefore produce larger savings than removing a few hundred input tokens.

Cache creation can also cost money. Anthropic charges $3.75 per million tokens for a five-minute cache write and $6 for a one-hour write; corresponding cache reads are $0.30 and $0.60. A longer retention period is worthwhile only when the same prefix is reused enough to offset that write cost.

The operational rule is simple: treat context as inventory, cache the stable inventory, and trim the volatile inventory before every request.

Key Concepts, Explained Simply

Prompt caching reuses an identical prompt prefix. The provider stores or retains that prefix and bills later requests at a lower cached-input rate. It is not semantic caching: changing the system prompt, tool order, or leading documentation can prevent a hit.

A context window is the total token budget available for instructions, retrieved documents, tool definitions, conversation history, and generated output. Some providers also impose a separate maximum-output limit, so a nominally large window does not guarantee a large usable answer.

That budget is also temporary: it resets with every new session. Persisting context across conversations needs a separate memory layer, covered in AI Memory Systems Explained 2026.

Context trimming removes or compresses content that is irrelevant to the current request. Examples include deduplicating retrieved passages, summarizing old conversation turns, stripping document metadata, and excluding tools the request will not use.

Structured output constrains a response to a defined JSON schema. Although it avoids reducing token consumption by itself, it can eliminate validation failures and automatic retries, which are often more expensive than the schema itself.

Here is a compact operational reference for the services discussed below:

Service Caching control Context planning note Main cost consideration
OpenAI API Automatic prefix caching GPT-5 family: 400,000 tokens Cached reads receive a large discount
Anthropic API Explicit cache_control Sonnet 4.5: 1 million tokens Cache-write and cache-read fees differ
Google Gemini API Automatic or explicit cachedContents Gemini 2.5 Pro: 1,048,576 tokens Explicit caching can add storage charges
Azure AI Foundry Model and deployment dependent Check the deployed model card Use the platform’s native model billing
Amazon Bedrock Model and region dependent Check supported cache fields Verify availability in the selected region
Google Vertex AI Gemini and partner-model support Governance and deployment dependent Cache retention can affect storage cost

Cloud platforms such as Azure AI Foundry, Amazon Bedrock, and Vertex AI do not create a universal cache layer. Caching behavior follows the selected underlying model and the platform’s implementation.

A Practical Token-Optimization Workflow

Step 1: Establish a Cost-Per-Answer Baseline

Begin by measuring requests instead of estimating them. Record the model, endpoint, input tokens, cached input tokens, output tokens, retries, latency, and business outcome for every request.

Normalize provider-specific usage fields into one internal event. For example:

{
  "request_id": "req_01J8N4",
  "model": "gpt-5-mini",
  "task": "support_triage",
  "input_tokens": 18432,
  "cached_input_tokens": 12288,
  "output_tokens": 742,
  "latency_ms": 1180,
  "accepted": true
}

OpenAI exposes cached tokens through usage.prompt_tokens_details.cached_tokens. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens. Gemini reports cachedContentTokenCount.

Segment the results by workflow—such as chat, RAG, classification, and code generation—because an aggregate cost figure can hide an expensive route. Track cost per accepted answer, not merely cost per request. Otherwise, a cheap response that fails validation and triggers three retries may appear cheaper than a successful response that costs more once.

Step 2: Build a Stable, Cacheable Prefix

Arrange each request so that predictable content comes first and changing content comes last. A useful order begins with fixed system or developer instructions, followed by stable tool definitions and output schemas. Next comes static documentation or reference material, concluding with dynamic retrieved passages and the current user request.

Keep tool order and JSON schema formatting identical across requests. Timestamps, random IDs, request metadata, and user-specific values near the beginning of the prompt can invalidate the entire cache prefix.

OpenAI caches eligible prefixes automatically. For GPT-5-family models, the prompt must also meet the model-specific minimum prefix length of 1,024 tokens. A stable cache-routing key can help requests reach the same cache infrastructure:

response = client.responses.create(
    model="gpt-5-mini",
    input=[
        {"role": "system", "content": STABLE_SYSTEM_PROMPT},
        {"role": "user", "content": user_question},
    ],
    extra_body={"prompt_cache_key": "support-v3"},
)

Anthropic requires an explicit cache marker:

response = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=700,
    system=[{
        "type": "text",
        "text": STABLE_SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral", "ttl": "1h"},
    }],
    messages=[{"role": "user", "content": user_question}],
)

Use Anthropic’s five-minute TTL for short bursts and the one-hour TTL only when the same prefix will be reused repeatedly.

Gemini supports explicit cached-content resources:

cache = client.caches.create(
    model="models/gemini-2.5-flash",
    config={
        "ttl": "3600s",
        "system_instruction": {
            "parts": [{"text": STABLE_SYSTEM_PROMPT}]
        },
    },
)

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=user_question,
    config={"cached_content": cache.name},
)

Choose cache TTLs from observed reuse patterns rather than maximum retention. An unused cache still consumes storage or creates write charges without delivering the discounted read price.

Step 3: Trim Context by Value

Caching handles repeated content; trimming removes content the current answer does not need. Before calling the model, deduplicate retrieved chunks, remove irrelevant results, summarize older conversation turns, and exclude unused tool definitions.

Set an operational context budget instead of relying on the model’s maximum window. A support RAG workflow might cap input at 12,000 tokens even when the selected model supports 1 million. Reserve room for the response and a safety margin:

usable input budget = operational context cap
                     - maximum output
                     - safety margin

A simple selection pass can enforce that limit:

def pack_context(chunks, budget=12_000):
    kept, used = [], 0

    for chunk in chunks:  # Already ranked by relevance.
        size = estimate_tokens(chunk["text"])
        if used + size > budget:
            continue
        kept.append(chunk)
        used += size

    return kept

Token estimates are sufficient for budgeting, but validate the final result using provider usage data. Keep the current question, the highest-ranked evidence, required citations, and the output contract. Summarize old dialogue rather than deleting it indiscriminately, and place the decisive task instruction next to the current request when models perform better with late instructions.

The stable system prompt should also be treated as an interface, not rewritten for every request. A fixed role, goal, constraints, and output-contract structure—the approach described in the AI Director Mode solution—makes prompt versions, cache invalidation, and regression testing easier to manage.

Step 4: Enforce Structured Output and Route Models

Define the expected response once and enforce it at the API level. OpenAI supports strict JSON Schema through the Responses API:

response = client.responses.create(
    model="gpt-5-mini",
    input=messages,
    text={
        "format": {
            "type": "json_schema",
            "name": "ticket",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "category": {"type": "string"},
                    "priority": {"enum": ["low", "high"]}
                },
                "required": ["category", "priority"],
                "additionalProperties": False
            },
        }
    },
)

Anthropic does not require a separate JSON-mode flag for every use case. Define a tool with an input_schema and use forced tool choice when the result must be machine-readable. Gemini supports response_mime_type="application/json" alongside response_schema in GenerateContentConfig.

Then route tasks according to difficulty. Use GPT-5 mini or Gemini 2.5 Flash for extraction, classification, routing, and routine summaries. Reserve GPT-5 or Claude Sonnet 4.5 for complicated reasoning, but evaluate both on a representative test set before changing models.

Set a deliberate max_tokens limit. For reasoning models, also control the reasoning budget because internal reasoning tokens can consume part of the output allowance. Use OpenAI’s Batch API or Anthropic’s Batch API for non-interactive workloads, but do not sacrifice interaction speed where users are waiting.

Finally, compare solutions using quality-adjusted cost:

cost per accepted answer = total request and retry cost
                           ÷ number of answers passing evaluation

This prevents a cheaper model with a higher retry or correction rate from appearing economical.

Common Mistakes & How to Avoid Them

Putting volatile content at the front is a significant error. A timestamp, conversation ID, or randomized instruction before the stable prompt changes the cache key. Append changing values after the reusable prefix and keep tool ordering deterministic.

Keeping a cache longer than it is reused wastes resources. A one-hour Anthropic cache or Google cached-content object may cost more than it saves if only two requests use it. Review cache creation counts and cache-hit ratios weekly, then align TTLs with actual traffic.

Trimming context without measuring quality can silently damage the product. Compare the original and trimmed prompts on representative tasks, including citation accuracy, refusal behavior, and edge cases. Keep the full context as a fallback for low-confidence cases.

Measuring only token price is flawed. Output tokens, retries, tool calls, latency, and human correction all affect the real cost. Route difficult requests to a stronger model when a smaller model’s accepted-answer rate declines. For the wider spending picture beyond API tokens, including GPU costs and the fees that hide in vendor invoices, see our AI agent costs in 2026 guide.

Cost & Effort Reality Check

Consider a representative GPT-5 mini workload: 10,000 monthly requests, each containing 10,000 input tokens and generating 1,000 output tokens. That produces 100 million input tokens and 10 million output tokens.

Without caching, input costs $25 and output costs $20, for a total of $45 per month. If 70% of input tokens receive cache hits, the request costs change. Cached input costs $1.75, uncached input costs $7.50, and output remains $20, resulting in a total of $29.25, or 35% less. If context trimming then removes 40% of input while maintaining the same cache-hit rate, the adjusted costs are $1.05 for cached input, $4.50 for uncached input, and $20 for output. This produces a total of $25.55, which is 43.2% below the original $45.

This example assumes OpenAI’s automatic cache, unchanged quality, and no additional workload growth. Anthropic cache-write charges and Google’s explicit-cache storage costs must be added to their respective calculations.

Engineering effort also matters. A realistic small-team implementation requires approximately one to two days for usage logging and attribution, one to three days for stable-prefix and cache configuration, and three to five days for trimming, schemas, and quality evaluation. That is roughly five to ten engineer-days initially, followed by perhaps one or two hours of weekly monitoring.

At a $45 monthly bill, this work will not pay back. At a $5,000 monthly bill, the same 43.2% reduction would save about $2,160 per month, making a one- or two-week implementation economically straightforward.

Getting Started This Week

The first step is to measure one representative workflow. Select 50 to 100 real requests, capture input, cached input, output, retries, latency, and acceptance, and calculate the cost per accepted answer. Record the model, prompt version, and cache configuration alongside each request.

The second step is to cache one stable prefix. Move timestamps and request-specific fields to the end, then enable OpenAI automatic caching, Anthropic cache_control, or a Gemini cachedContents resource. Start with five-minute Anthropic TTLs and extend them only after observing repeated use.

The third step is to trim and evaluate before expanding the change. Add a token budget, deduplicate retrieved passages, enforce a JSON schema, and test the modified prompt against the original set. Roll it out gradually to 5–10% of traffic, then compare cache-hit rate, accepted-answer cost, and quality before completing a wider rollout.

If a no-code agent platform makes the API calls, verify whether it exposes provider-native caching and usage fields. The best no-code AI agent builders 2026 comparison can help with that due-diligence, but token billing and cache controls still operate at the provider API boundary.

FAQ

Should I use prompt caching or trim the context?

Use both. Caching makes repeated stable prefixes cheaper, while trimming removes irrelevant or duplicated content from every request. Caching a bloated prompt saves money compared with sending that same bloated prompt uncached; however, it still incurs the cost of processing the cached tokens.

Does prompt caching change the model’s answer?

Prompt caching does not change the model or its weights, so a cache hit normally produces the same output distribution as the identical uncached request. It does not guarantee deterministic answers, and stale cached instructions can produce stale results when documentation changes.

How long does a prompt need to be before caching works?

The threshold depends on the provider and model. OpenAI’s GPT-5 family requires at least 1,024 tokens for automatic prefix caching, while Anthropic publishes model-specific minimums that are commonly 1,024 tokens for current Sonnet-class models. Gemini’s explicit cached-content resources can be more useful for large static documents than for short prompts.

Should a small team use a million-token context window?

Only if the task genuinely needs it. Set an operational token budget based on relevance, output requirements, and evaluation results. A 12,000-token curated context can outperform a much larger prompt, cost less, and make the application easier to maintain.

Use a three-layer policy in 2026: cache stable prefixes, trim volatile context, and route each request to the cheapest model that still passes your quality checks. Measure cost per accepted answer, review cache-hit ratios, and treat token optimization as an ongoing engineering practice rather than a one-time prompt edit.

Disclosure: This article may contain affiliate links. We may earn a commission at no extra cost to you.

Last Updated: November 1, 2026 | Specs and prices subject to change. Please verify current pricing on Amazon.

1 thought on “Prompt Caching and Context Windows in 2026: Cut Costs”

  1. Pingback: AI Memory Systems Explained 2026: Long-Term Context

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top