One line, a third off: we measured the prompt cache on a paid key

We ran the same agent on a paid OpenAI key, twice: once as it was, once with a cache key.

• 4 runs on gpt-4o-mini, each starting with the same 1,964-token opening.
• Without a cache key: 0 tokens read from cache.
• With withCacheBreakpoints(model, { cacheKey }): runs 2 to 4 each read 1,792 of those 1,964 tokens from cache.
• Each of those calls cost 44% less.
• The four runs together cost 33% less. The first run pays full price, because nothing is in the cache yet.
• The cost GNL computed from its journal was equal to the cost we computed separately from OpenAI's raw usage numbers.

The change was one line of code.

The optimizer's diagnosis, diagnoseAgent, reads an agent's cost from what the provider billed, as recorded in the agent's own journal. It calls no model. For the cache, it counted one thing: the opening a run re-sends on each of its later steps. That number comes straight from the bill.

A saving on paper is still a guess. So we paid for real calls and read the bill.

The measurement found a saving the diagnosis did not see. The diagnosis only counted the opening re-sent within one run. The saving we measured came from the same instructions re-sent run after run.

So we fixed the diagnosis. It now also counts the same instructions re-sent by runs that start within 5 minutes of one another; a provider keeps a cache for a few minutes. This number is marked as an estimate, because the journal keeps the instructions, not their token count.

The test for the fix uses the measured numbers. When the later runs re-send the instructions and read nothing from cache, the diagnosis reports a cache finding. When each later run already reads 1,792 tokens from cache, it reports nothing left to save.

A provider reads from cache only when the start of a request matches an earlier one. We captured the request bodies locally and compared them across runs. GNL adds nothing that varies between runs. What your agent sends the same way, the provider sees the same way.

The cache key does not change the tokens. It tells OpenAI which requests belong together. On Anthropic, the same wrapper places cache marks on the system message and the last message instead.

The whole experiment cost $0.0042: 66 calls on gpt-4o-mini.

We first picked gpt-4.1-nano. It was still listed, but already retired, so the runs went to gpt-4o-mini.

If your agent runs on OpenAI or Anthropic, wrap its model. On OpenAI, give it a key: name the agent, and the tenant too when tenants must not share a cache.

import { withCacheBreakpoints } from '@gnldev/optimizer';
import type { AgentConfig } from '@gnldev/durable';
import { openai } from '@ai-sdk/openai';

const agent: AgentConfig = {
  model: withCacheBreakpoints(openai('gpt-4.1'), { cacheKey: 'support' }),
};

To see first whether it pays for your agent, run the diagnosis on its journal. It reads what was billed and calls no model.

import { diagnoseAgent } from '@gnldev/optimizer';

const d = await diagnoseAgent({ journal: storage.runs, agent: 'support' });
for (const f of d.findings) console.log(f.lever, f.tokens, f.estimatedSavingUsd?.toFixed(2) ?? 'unpriced', f.why);