Replay-based regression

A trace tells you what happened once. It cannot tell you what would happen if you changed the model — for that you have to run the same input again. Because the journal froze the run's input, replayRun can do exactly that: it reads the recorded input and runs it fresh under a new model, prompt, tool set or guard.

diffRuns then compares the two runs at their decision points — every model step and every tool call. Alignment is not by toolCallId (the model and SDK assign those randomly, so two independent runs never share them) but by the model step plus the content order of the tool calls it produced. That is what makes two separate runs positionally comparable.

The result names the first place the runs diverged (divergentAt), so "the new model behaves differently" becomes "the new model chose a different tool at step 2".

import { replayRun, diffRuns, regressionReport } from '@gnldev/durable';
import type { RunDiff, DiffEntry, DecisionPoint } from '@gnldev/durable';

Everything here lives in @gnldev/durable; no extra package is required. Scoring is deliberately not part of it — regressionReport accepts a scorer you supply, and the ready-made scorers live in @gnldev/evals.

Start from a run that is already in the journal. replayRun reads its recorded input and runs it again under whatever you override — here, a different model. It writes under a new runId and never touches the records of the original run.

const { newRunId } = await replayRun({
  journal,
  runId: 'order-123',          // the recorded run
  model: anthropic('claude-opus-5'),
  tools: { chargeCard },
});
// the input is read from the journal — you do not pass the prompt again

Now diff the two. RunDiff.steps carries one entry per decision point with a status of same, changed, missing or added; summary counts them, and divergentAt is the index of the first difference.

const diff = await diffRuns(journal, 'order-123', newRunId);

diff.summary;      // { same: 3, changed: 1, missing: 0, added: 0 }
diff.divergentAt;  // 3 — the index of the first decision that differs

For a model decision the detail carries both sides' text and their tool-call lists (name + argsHash); for a tool decision it carries the tool name, the argument hashes, the status and both outputs. A structural mismatch (a model step where the other run has a tool call) is reported in note.

const entry: DiffEntry = diff.steps[diff.divergentAt!];
// {
//   step: 1,
//   kind: 'tool',
//   status: 'changed',
//   detail: {
//     toolName: 'chargeCard',
//     argsHashA: '9f21…', argsHashB: 'c704…',
//     statusA: 'succeeded', statusB: 'succeeded',
//     outputA: { charged: 500 }, outputB: { charged: 5000 },
//   },
// }

regressionReport is a thin wrapper: it diffs the two runs and, if you pass a scorer, runs it on the diff. It contains no scoring logic of its own — the dependency points one way, from @gnldev/evals to @gnldev/durable, never back.

const report = await regressionReport(journal, 'order-123', newRunId, {
  scorer: (d) => d.summary.changed / d.steps.length,
});
// { baseRunId, newRunId, diff, score }

When an agent answers using something memory recalled, "the model must have read it from the recall snippet" is an inference, not a measurement. stripMemoryContext turns it into an experiment: it reads the run's :memctx provenance record, keeps only the turn's own incoming messages, drops everything memory composed in front of them, and re-asks. Then you diff the two answers.

const withoutMemory = await replayRun({
  journal,
  runId: 'chat-42',
  model,
  stripMemoryContext: true,   // keep only this turn's own messages
});

const diff = await diffRuns(journal, 'chat-42', withoutMemory.newRunId);
// same answer  -> the recall was not what produced it
// different    -> the injected context carried the answer

Two honest boundaries are written into the code rather than glossed over. It throws when the run has no usable :memctx record — a run from before provenance existed cannot be stripped honestly. And the frozen system string is not surgically edited: if working memory was injected there, it stays, and the code says callers should disclose that.