Cost optimizer

An agent's money goes to five places: the same opening (system prompt, tool definitions) re-sent on every step; old tool output carried along after it was used; the expensive model doing simple work; non-urgent work at interactive prices; answers longer than they need to be. Most of it is invisible from your code, because each step looks reasonable on its own.

The journal already has the answer. Every model step it records carries what the provider billed: input tokens, output tokens, tokens read from cache. @gnldev/optimizer reads that record, says which of these your agent pays for, gives you the lever for each, and lets you prove a change keeps the work the same before you make it.

npm install @gnldev/optimizer

diagnoseAgent reads an agent's latest completed runs and adds up cost, tokens, cache reads and cache writes, per model. No model is called.

import { diagnoseAgent } from '@gnldev/optimizer';
import { SqliteStorage } from '@gnldev/durable/sqlite';

const storage = new SqliteStorage('runs.db');

// Where the money went — read from what the provider billed. No model is called.
const d = await diagnoseAgent({ journal: storage.runs, agent: 'support' });
for (const f of d.findings) console.log(f.lever, f.tokens, f.estimatedSavingUsd?.toFixed(2) ?? 'unpriced', f.why);

• The opening re-sent within a run. Every step after the first re-sends the run's opening. The first step's input tokens are that opening, so this number comes straight from the bill. What the provider already read from cache is subtracted; what is left is what a cache could still save, after one cache write per run.
• The same instructions re-sent by runs that start within 5 minutes of one another. A provider keeps a cache for a few minutes, so a run that starts soon after another with the same system prompt could read it from cache. This one is an estimate: the journal keeps the instructions, not their token count.
• Old tool output. A tool result rides along in every later step. Results older than keepLast (default 2) steps are counted. Also an estimate (4 characters to a token), and the diagnosis says so.

A provider does not cache a short prefix. minCacheTokens (default 1024) is the smallest opening counted as cacheable — OpenAI's floor and Anthropic's for most models. A shorter one is never counted as a saving; the diagnosis adds a note instead.

Findings come sorted by estimated saving. Each one says the number behind it in one sentence. Prices come from @gnldev/durable's price table; a model missing from it is reported as unpriced, never as free.

Two changes act on the findings. Both change what the provider is sent for a step, never what the journal records, and both are pure functions of the request, so a resumed run asks the same thing.

withCacheBreakpoints(model, { cacheKey, ttl }) (and the middleware cacheBreakpoints) asks each provider for its prompt cache. Anthropic: cache marks on the system message and the last message, within its limit of 4. OpenAI: your cacheKey is sent as promptCacheKey; without a cacheKey no key is sent. Other providers: untouched. Quality cannot change — the provider is sent the same tokens.

A cache write costs more than plain input on Anthropic: 1.25x for a ttl of 5 minutes (the default), 2x for 1 hour. A run of one step pays for a cache it never reads; the diagnosis says whether an agent's runs are long enough for it to pay.

maskOldToolOutputs replaces tool output older than keepLast (default 2) steps with a one-line note, for that step only. A result shorter than minChars (default 200) characters is kept whatever its age. The journal keeps the full output. This one can change what the agent does — replay it first.

import { maskOldToolOutputs, withCacheBreakpoints } from '@gnldev/optimizer';
import type { AgentConfig } from '@gnldev/durable';
import { openai } from '@ai-sdk/openai';

const agent: AgentConfig = {
  model: withCacheBreakpoints(openai('gpt-4.1'), { cacheKey: 'support' }),
  processors: [maskOldToolOutputs({ keepLast: 2 })],
};

replayCounterfactual plays a recorded run again with a candidate: another model, system prompt or context policy (prepareStep). The question is not "is the cheaper model good", but "would it have done the same work, on this conversation". The replay itself lives in @gnldev/experiment (moved there from the optimizer in 0.16, with no alias left behind); it is installed with @gnldev/optimizer.

import { replayCounterfactual } from '@gnldev/experiment'; // installed with @gnldev/optimizer
import { SqliteStorage } from '@gnldev/durable/sqlite';
import { openai } from '@ai-sdk/openai';

const storage = new SqliteStorage('runs.db');

// Would a cheaper model have done the same work? Replay a recorded run — no tool is ever executed.
declare const tools: Parameters<typeof replayCounterfactual>[0]['tools']; // the agent's own tool definitions
const r = await replayCounterfactual({ journal: storage.runs, runId: 'order-42', model: openai('gpt-4.1-mini'), tools });
console.log(r.divergence ?? 'same calls', r.recorded.costUsd, r.candidate.costUsd);

No tool ever runs. Each tool's execute is replaced by a lookup in the recording: a call the recording made (same tool, same arguments) gets the recorded result. A call the recording never made has no result to give, so the replay stops there and reports a divergence. The result has both sides' cost and calls, the calls the candidate skipped, and whether it made them in another order.

replayAgent does the same for an agent's latest runs under one candidate and sums them up: divergence rate, runs with a skipped call, runs with reordered calls, cost ratio. When a model is unpriced, no ratio is made up. A run that could not be replayed (no recorded input, a model error, a timeout) is listed apart and counts toward nothing.

DEFAULT_SUITE is 26 scenarios a tool-using agent does: refunds, lookups, multi-step work, plain conversation, tricky arguments. Each has fake tools over one small fake store, and an expectation a program checks: calls that must be made, calls that must not, facts the answer must state. No judge model decides. DEFAULT_SUITE and runSuite are in @gnldev/experiment too; compareOnSuite stays in the optimizer.

runSuite runs it with one model; compareOnSuite runs it with two and reports the cost ratio, the regressions and the improvements. A model that could not be asked — a 404, a rate limit, a timeout — marks the scenario error. Errors are kept apart from quality failures: they are left out of the pass rate and out of the comparison.

saveExperiment keeps a replay's outcome in the journal, and listExperiments reads them back, newest first. A record holds run ids, calls and numbers — no conversation text. So a trend stays visible: a candidate that diverged in 8% of runs last week and 3% this week.

renderReport turns the diagnosis and the kept replays into a Markdown report: where the money goes, the findings, the replays tried, and a recommendation. It recommends; it changes nothing.

Give Studio the optimizer and it shows an Optimizer tab: the diagnosis per agent, a replay of an agent's latest runs with a candidate model you offer, the replays tried so far, and a downloadable report.

import { studioOptimizer } from '@gnldev/optimizer';
import { createStudioApp } from '@gnldev/studio';
import { toJournal, type CreateGnlConfig } from '@gnldev/durable';
import { SqliteStorage } from '@gnldev/durable/sqlite';
import { openai } from '@ai-sdk/openai';

declare const config: CreateGnlConfig; // the config createGnl is given
const storage = new SqliteStorage('runs.db');

const studio = createStudioApp({
  reader: toJournal(storage),
  apiBase: '/studio',
  optimizer: studioOptimizer({ config, models: { 'gpt-4.1-mini': openai('gpt-4.1-mini') } }),
});

studioOptimizer({ config, models, pricing }) takes the config createGnl is given (its agents' tools answer a replay), the candidate models by the name Studio shows, and optional prices. Studio hands every call the reader of the request it serves, scoped to the caller's organization, and never holds a model itself. A replay calls a model, so it needs write access and is written to the audit log. Studio is a fetch handler: in a Hono app, mount it with app.mount(...).

Per-request routing between models, escalating to a bigger model mid-run, answering from a "similar question" cache, compressing tool arguments or results. Each one can change what a run does without anyone seeing it: a routed run loses its provider cache, and a cached answer skips tool calls that should have happened.

A run's model never changes mid-run. A cheaper model is a change you make to an agent, for its new runs.