Low prompt cache hit rate on OpenAI and Anthropic: causes and how to calculate it

A prompt cache can only reuse the exact beginning of a request. When your prompt cache hit rate is low, either something near the start changed between calls, such as the instructions, the tool list, a request setting, or an earlier turn that got rewritten, or the next call arrived after the cache entry had expired. Everything after the first changed token is processed again at full price, while the unchanged part before it can still be read from cache.

Check the arithmetic before hunting for either cause. On Anthropic and on the AIVAX gateway the total input is the sum of several fields; apply that sum to OpenAI usage, where prompt_tokens already includes the cached tokens, and the hit rate can never exceed 50%. Skip it on AIVAX usage and you get numbers above 2,000%. A wrong denominator looks exactly like a caching problem, so check the formula before touching the prompt.

Calculate the hit rate with each API's own definition of input tokens

Divide cached input tokens by total input tokens, and sum both over the requests you care about: a session, a user, a day. A ratio of sums tells you more than an average of per-request ratios, because one short request with a 0% rate should not weigh as much as a 100,000-token one. The hard part is the denominator, because each API defines "input tokens" differently.

The OpenAI guide defines the rate as total cached tokens divided by total input tokens, and its Responses usage fields don't overlap: cached_tokens excludes every token charged as a write, including tokens upgraded from 30-minute to 12-hour coverage. Anthropic's tracking section gives the total as the sum of its three fields. If you divide cache_read_input_tokens by input_tokens alone, you are dividing by the uncached tail.

The AIVAX gateway returns an OpenAI-shaped usage object with a different prompt_tokens. In the gateway code, prompt_tokens is the provider's prompt count minus cached tokens, and total_tokens still includes the cached ones. Take a response with prompt_tokens: 84, cached_tokens: 1792, completion_tokens: 16, and total_tokens: 1892. Its hit rate is 1792 / (84 + 1792), about 95.5%. Divide by prompt_tokens and you get 2,133%, which tells you the OpenAI formula went onto the wrong shape. The gateway passes on only the read count. Provider cache-write counts don't appear in its usage object.

In the gateway code, one request that runs a tool loop makes several model calls, and the usage it reports is the sum of their usage; billing charges cached tokens at the model's cached-input price, or bills everything as input when the model has none. Model rates are in the pricing docs, and pricing each AIVAX call shows how the Consumed-Credits header and the provider field turn that usage into a cost.

This script reads one usage object per line and prints the aggregate rate and the rate for each request. Save it as hit-rate.mjs and run node hit-rate.mjs aivax usage.jsonl.

import { readFileSync } from "node:fs";

const [shape, file] = process.argv.slice(2);

const readers = {
  "openai-responses": (u) => ({
    cached: u.input_tokens_details?.cached_tokens ?? 0,
    total: u.input_tokens,
  }),
  "openai-chat": (u) => ({
    cached: u.prompt_tokens_details?.cached_tokens ?? 0,
    total: u.prompt_tokens,
  }),
  anthropic: (u) => {
    const cached = u.cache_read_input_tokens ?? 0;
    return {
      cached,
      total: cached + (u.cache_creation_input_tokens ?? 0) + u.input_tokens,
    };
  },
  aivax: (u) => {
    const cached = u.prompt_tokens_details?.cached_tokens ?? 0;
    return { cached, total: u.prompt_tokens + cached };
  },
};

const read = readers[shape];
if (!read || !file) {
  console.error("usage: node hit-rate.mjs <openai-responses|openai-chat|anthropic|aivax> usage.jsonl");
  process.exit(1);
}

const rows = readFileSync(file, "utf8")
  .split("\n")
  .filter(Boolean)
  .map((line) => read(JSON.parse(line)));

const pct = (cached, total) => (total ? ((100 * cached) / total).toFixed(1) : "0.0");
const cached = rows.reduce((n, r) => n + r.cached, 0);
const total = rows.reduce((n, r) => n + r.total, 0);

console.log(`${rows.length} requests: ${cached} of ${total} input tokens cached (${pct(cached, total)}%)`);
console.log(`per request: ${rows.map((r) => `${pct(r.cached, r.total)}%`).join(" ")}`);

Read the per-request line before the aggregate. A session that sits at 90% and drops to 0% on one request points to a specific change in that request. A rate that stays low from the first call points to the design: a dynamic value at the top, or a prefix below the minimum cacheable length.

The prefix breaks at the first token that changes

Both providers render the request in a fixed order and match it from the start. Anthropic documents the order as tools, then system, then messages, and a change at one level invalidates that level and every level after it. OpenAI caches the full rendered context, which covers hidden instructions, tool definitions, developer messages, and history, and lists the request settings that affect the cached prefix.

Length matters before any of this. OpenAI caches nothing below 1,024 visible input tokens on GPT-5.6 and later, and hidden system tokens don't count toward that minimum. On earlier models the minimum varies with tools, images, output schemas, reasoning effort, and verbosity (summary of model differences).

Anthropic's minimum depends on the model, from 512 tokens on the Claude 5.5 family up to 4,096 on Opus 4.5, Opus 4.6, and Haiku 4.5 (the full list). A shorter prompt is processed without caching and returns no error, so the only sign is cache read and write counts that stay at zero.

OpenAI points out that a shared prefix is not always a cached prefix: with a static developer message followed by a changing user message, implicit mode writes the cache through the dynamic content. The next request doesn't match that longer entry, and nothing marks the end of the static part as reusable on its own.

Anthropic has a separate limit: its lookback checks at most 20 positions back from a breakpoint, and it finds only entries that earlier requests wrote at their own breakpoints. If a conversation grows by 20 or more blocks between requests, the automatic breakpoint can miss the previous write. In both cases the fix is the same: put a breakpoint on the last block that stays identical across requests, and add a second one closer to the end when the history grows fast.

OpenAI and Anthropic can now name the field that changed

For a long agent session, what you want to know is which part of the request differed from the previous call. Both providers now answer that directly.

On GPT-5.6 and later in the Responses API, set prompt_cache_options.comparison_response_id to the id of an earlier response, and the new response carries prompt_cache_diagnostics. It returns cache_hit, cache_miss, comparison_response_not_found, or unavailable. A miss comes with a reason, such as tools_changed, context_compacted, input_changed, model_changed, or reasoning_effort_changed, plus an estimate of the tokens that were not reused. According to OpenAI's prompt cache diagnostics guide, the comparison costs nothing extra, is best effort, and works only while the earlier response's diagnostic record has not expired.

On Anthropic, send a diagnostics object on every turn: previous_message_id: null on the first and the previous response's id after that. The response's diagnostics.cache_miss_reason.type points to the first divergence: model_changed, system_changed, tools_changed, or messages_changed, with an estimate of the input tokens after it. previous_message_not_found and unavailable mean no comparison was made, and they say nothing about whether your request changed. Cache diagnostics keeps a fingerprint of hashes and token estimates, never the prompt text, and only for requests that opt in.

The AIVAX gateway does not ask the provider for either diagnostic. Through the gateway, and on Chat Completions in general, you need a check that doesn't depend on the provider. If you capture your outgoing request bodies, this script compares two of them, reports the request-level settings that differ, and gives the index of the first message that changed. Those files can contain private messages, tool results, or credentials pasted into prompts, so store them as sensitive data. Save the script as first-break.mjs and run node first-break.mjs previous.json current.json.

import { readFileSync } from "node:fs";

const [previous, current] = process.argv.slice(2).map((f) => JSON.parse(readFileSync(f, "utf8")));
const same = (x, y) => JSON.stringify(x) === JSON.stringify(y);

const settings = [
  "model", "tools", "tool_choice", "parallel_tool_calls", "response_format",
  "text", "reasoning", "reasoning_effort", "verbosity", "thinking", "system", "instructions",
];

for (const key of settings) {
  if (!same(previous[key], current[key])) console.log(`differs before the conversation: ${key}`);
}

const before = previous.messages ?? previous.input ?? [];
const after = current.messages ?? current.input ?? [];
const shared = Math.min(before.length, after.length);

let i = 0;
while (i < shared && same(before[i], after[i])) i++;

console.log(
  i === shared
    ? `first ${shared} items identical (${before.length} before, ${after.length} now)`
    : `first changed item: index ${i} (role: ${after[i].role})`,
);

Index 0 with a system or developer role almost always means dynamic text in the instructions. A differs before the conversation line for tools means the list changed, and nothing after it could be reused. Identical items rule out only the fields the script checks. It cannot see history that the server rebuilds from previous_response_id, and on the AIVAX gateway it sees your request, not the one the gateway sends upstream, which the AIVAX sections below cover.

Compaction lowers the hit rate and can still lower the bill

Compaction replaces earlier turns with a shorter version, so the prefix changes from the first replaced token onward. OpenAI's guide says that the first request after compaction may reuse less of the previous cache, and context_compacted is one of its diagnostic reasons. It also says to compare total input cost before and after, because fewer input tokens can cost less even when the hit rate falls.

I agree with that test, and the arithmetic is quick. Assume, for illustration only, that cached tokens cost 10% of the regular input price, and ignore cache-write charges. A request with 150,000 input tokens at a 90% hit rate costs the equivalent of 150,000 × (0.9 × 0.1 + 0.1) = 28,500 regular tokens. After compaction, 20,000 tokens at a 40% hit rate cost 20,000 × (0.4 × 0.1 + 0.6) = 12,800. The hit rate dropped from 90% to 40%, and this illustrative input cost dropped by 55%. That leaves out the cost of producing the compacted context and the output. Real read and write prices differ by model, so run the numbers with your provider's price list.

The compaction in this section happens in your client or in the provider's API. The AIVAX gateway doesn't compact: when a conversation exceeds ContextMaximumSize, its overflow actions are Throw, Truncate, and TruncateHard. Wherever compaction runs, keep the system prompt and tool definitions byte-identical, and let later turns build on the compacted context instead of compacting again on every request.

AIVAX fixed two prefix breakers in October 2026

An AIVAX AI Gateway builds the upstream request for you, so part of the prefix is out of your hands. On 9–10 October 2026, AIVAX changed two gateway behaviors that put dynamic content at the start of that request; both are in the changelog under Saturday, October 10th.

The hosted web chat used to add a system message with the user's local date and time zone at the start of each conversation. It no longer does. A gateway that needs the date enables the DateTime built-in tool, and the date enters the history as a tool result when the model asks for it.

The other fix was in how the gateway combined its system instructions: the parts were joined in the order their sources finished loading, so two identical requests could produce different instructions. They now follow the declared order: the gateway's own instruction first, then MCP server instructions, remote instruction sources in their configured order, and anything added during the request. If your traces show the system prompt changing between turns before those dates, these two are the first suspects.

Memory was handled in the same round. The gateway no longer appends saved memories or reminders to the system instructions, so saving a memory leaves the prefix alone; the model retrieves memories with memory_search, and the result enters the conversation as a tool response. How agent memory works covers the write path.

Gateway settings that still rewrite the prefix

The settings below control context size, and each changes what the gateway sends upstream without appearing in your own request body.

A web chat session can set contextLocation, a URL that AIVAX loads during response generation and appends to the session context. The chat client docs describe it as context "that may change over time". In the code, the URL is fetched on every request with no cache, and the text goes into the system instructions. If that endpoint returns anything that varies per call, such as a timestamp or a reordered list, the system prompt changes and everything after it misses. I would make that endpoint deterministic for a given session state before looking anywhere else.

ToolContextCount keeps only the last N tool responses and replaces older ones with [tool response truncated - call this tool again] on every request. Once the history holds more than N tool responses, each new one turns another kept response into the placeholder, which edits a message in the middle of the history, so by the exact-prefix rule the part after it cannot reuse the previous entry. The pipeline docs now point this out. With the Truncate overflow action, the gateway removes the oldest non-system messages until the conversation fits, which moves the start of the history. A gateway defaults to Throw.

With HideToolsWithoutSkill, the gateway filters the tool list by skill visibility, and if the filtered list differs between two requests, so does the prefix. The MCP tool-definition guide explains the flag.

A gateway that uses the complexity router classifies the latest user request as low, medium, or high complexity and picks the model and reasoning effort configured for that level, and it reports the level in the X-Model-Routed-Complexity response header. A conversation that moves between levels moves between models, and every model has its own cache. Route by complexity, price by the task covers when that trade is worth it.

On cache controls, the gateway does less than a direct provider call. It forwards prompt_cache_key on the Chat Completions transport. In the gateway code, a BYOK gateway configured for the OpenAI Responses transport drops the field during translation. OpenAI documents two roles for the key: on GPT-5.6 and later it separates cache accounting and is not needed to optimize caching, and on earlier models a stable key helps route requests that share a prefix. The gateway has no handling for Anthropic cache_control, and it doesn't set prompt_cache_options.ttl or prompt_cache_retention, so the provider's default lifetime applies.

If your numbers show a loss, I would try the conservative changes first: raise ToolContextCount or leave it unset for short tool loops, keep one model per conversation, and make contextLocation stable. Measure each change against your own usage logs. The learn guide on caching separates prompt caching from response caching, and the inference reference lists the request parameters.

Cache lifetime causes misses that a stable prefix cannot fix

An expired entry is a miss no matter how clean the prefix is. On GPT-5.6 and later in the Responses API, prompt_cache_options.ttl accepts "30m", the default, or "12h". Both are minimum lifetimes, a read extends coverage to at least 30 minutes after it, and the 12-hour option has a higher write rate with the same read rate. Earlier models use prompt_cache_retention: in_memory usually keeps entries for 5 to 10 minutes of inactivity, up to an hour, and 24h keeps them up to 24 hours on the models that support it. The default depends on your organization's data retention setting: 24h without Zero Data Retention and in_memory with it.

On Anthropic, an entry lasts 5 minutes by default and is refreshed at no charge each time it is used, or one hour with "ttl": "1h" at a higher write price (1-hour cache duration).

The choice depends on how far apart your turns are. If users read and think for longer than the window before replying, a longer lifetime can cost less than writing the cache again on every turn. If turns arrive every few seconds, the default window already covers you and the cause is elsewhere.

On OpenAI, volume is a factor too: cached state lives on individual machines, and traffic above 15 requests per minute can lead to overflow routing to machines that don't hold the entry. A higher hit rate makes requests cheaper and faster but frees no rate-limit headroom: cached input tokens still count toward tokens-per-minute limits.

Read on 26 October 2026: OpenAI and Anthropic behavior comes from their prompt caching and cache diagnostics pages as published that day. AIVAX behavior comes from the gateway source and the maintained docs, and the effect of each gateway setting is derived from how the gateway builds the upstream request. On November 10, 2026 I ran both scripts with Node 24.12.0 on synthetic input: the 84 / 1792 / 16 / 1892 usage line gives 95.5% with the aivax shape and 2,133.3% with the openai-chat shape, and the comparison script reported a changed tools setting and a first changed item at index 1. I also sent the same 3,913-token prompt three times to @openai/gpt-5.4-nano through the live gateway: the first call reported prompt_tokens: 3913 and cached_tokens: 0; the second and third reported prompt_tokens: 1097, cached_tokens: 2816 and a total_tokens of 3919, so the cached share is subtracted from prompt_tokens as described above, and the hit rate is 2816 / (1097 + 2816), about 72%. Caching on the native Anthropic path is described from the providers' documentation and the gateway source.