Evaluation: scorers & LLM-judge

@gnldev/evals scores an agent run's output from the journal trace (the recorded model/tool history) — it looks at the journal, not the live call. That's why scoring is deterministic and replayable: every time the same journal runs, rule-based scorers (exactMatch, contains, regex, embeddingSimilarity) give the same score; even an LLM-judge result invoked via scoreRun is memoized to the journal, so a resume doesn't make another model call — it returns the first score.

Typical scenario: run an agent over N test cases in a dataset, score each case, and take the average — that's what evalDataset is for. If the suite is called with a journal, each case is memoized durably: if the suite crashes partway through, completed cases don't re-run, the rest continue where they left off (a resumable eval suite).

The package has a peer-dependency on @gnldev/durable (for reading/writing the journal); LLM-judge also requires ai (the AI SDK).

pnpm add @gnldev/evals @gnldev/durable ai

import { scoreRun, exactMatch, contains, regexScore, embeddingSimilarity, llmJudge, evalDataset } from '@gnldev/evals';
import {
  faithfulness, hallucination, answerRelevancy, toxicity, bias, completeness, contextPrecision, toneConsistency,
} from '@gnldev/evals';
import { createDatasetsManager } from '@gnldev/evals';

1) Score a completed run — scoreRun reads the run's journal trace through a JournalReader, extracts the text of the last model entry, and applies the given scorers to that text:

import { scoreRun, exactMatch, llmJudge } from '@gnldev/evals';

const res = await scoreRun(reader, runId, [
  exactMatch(),
  llmJudge({ model, rubric: 'Is the answer correct and concise?' }),
], { expected: 'Paris' });

// res.output   -> the last model output in the journal
// res.scores   -> { 'exact-match': { score, reason }, 'llm-judge': { score, reason } }

If reader also satisfies the Journal interface (e.g. a SqliteStorage instance), scoreRun memoizes each scorer's result to the journal under the key `${runId}:proc:eval:${name}` — so if the same run is scored again (or resumed), even llmJudge doesn't make a second model call. You can turn this off with { memo: false }.

2) Batch evaluation over a dataset — evalDataset runs the agent on every case in the dataset, scores them, and totals the average per scorer:

import { evalDataset } from '@gnldev/evals';

const report = await evalDataset({
  dataset: { id: 'qa-suite', cases: [
    { id: 'c1', input: 'What is the capital of France?', expected: 'Paris' },
  ] },
  run: async (input, { runId }) => {
    // run the agent for this case; return { output } or a string
    return runAgent(input, { runId });
  },
  scorers: [exactMatch()],
  journal, // when given, case results are memoized -> a resumable suite
});

// report.aggregate -> { 'exact-match': 0.8 } (0..1 ortalama)

When the journal parameter is given, each case is memoized under `evalds:${dataset.id}` with the key `case:${caseId}`; if there's a crash partway through the suite, completed cases are returned from the journal instead of being re-run.

3) Rule-based scorers on their own — these functions, which implement the Scorer interface, can also be called directly without scoreRun/evalDataset:

const exact = exactMatch();
await exact.score({ output: 'Paris', expected: 'Paris' }); // { score: 1, reason: 'exact match' }

const emb = embeddingSimilarity(
  (text) => embed({ model: embeddingModel, value: text }).then((r) => r.embedding),
  { threshold: 0.85 },
);
await emb.score({ output: '...', expected: '...' });

4) Built-in scorers — the 8 built-in scorers are rubrics built on top of llmJudge (faithfulness, hallucination, answerRelevancy, toxicity, bias, completeness, contextPrecision, toneConsistency) — none of them call generateText directly, they wrap llmJudge; that's why scoreRun/dataset memoization comes for free.

import { faithfulness, hallucination, toxicity } from '@gnldev/evals';

const res = await scoreRun(reader, runId, [
  faithfulness({ model }),   // sample.context gerektirir — yoksa { score: 0, reason: 'context gerekli: ...' }
  hallucination({ model }),  // needs context; DIRECTION: 1.0 = NO hallucination (good)
  toxicity({ model }),       // needs neither context nor input; it scores the output alone
], { context: retrievedChunks });

Direction semantics (common to every scorer): a high score = a good result. This can be confusing for scorers whose name suggests the opposite (hallucination, toxicity, bias): e.g. a hallucination score of 1.0 means "NO hallucination", not "hallucination present".

Scorers that need context (faithfulness, hallucination, contextPrecision) require sample.context; those that need a question (answerRelevancy, completeness) require sample.input — if missing, they don't silently return 1.0, they return { score: 0, reason: '...' }. Every scorer also explicitly declares via sampleFields which extra fields (input/context) get added to the llmJudge prompt — toxicity/bias/toneConsistency deliberately use sampleFields: [] (so they aren't contaminated by irrelevant context — e.g. if the context is toxic, the output shouldn't get a falsely low toxicity score even if it's clean).

5) Dataset versioning + experiment comparison — createDatasetsManager adds a journal-based layer on top of evalDataset: it keeps a dataset's version history, records experiments idempotently, and compares two experiments (regression/improvement).

import { createDatasetsManager } from '@gnldev/evals';

const manager = createDatasetsManager(journal); // journal 'listKeys' desteklemeli

await manager.saveDataset(dataset);   // no new version is opened when the content is unchanged (hash comparison)

const exp1 = await manager.runExperiment({
  datasetId: 'qa-suite', run: runAgent, scorers: [exactMatch()], experimentId: 'baseline',
});
const exp2 = await manager.runExperiment({
  datasetId: 'qa-suite', run: runAgentV2, scorers: [exactMatch()], experimentId: 'candidate', label: 'yeni prompt',
});
// a second call with the same experimentId does NOT re-run evalDataset — it returns the recorded result (idempotent)

const diff = await manager.compare('qa-suite', 'baseline', 'candidate');
// diff.aggregate -> per scorer { baseline, candidate, delta }
// diff.regressions / diff.improvements -> how many cases got worse or better

Two families here never call a model. The four text scorers — contentSimilarity, keywordCoverage, textualDifference, answerSimilarity — are pure functions of output and expected: no cost, no latency, and no non-determinism to reason about. They are the right default for a large regression suite where an LLM judge would be both slow and expensive.

The trajectory scorers answer a different question: not "is the answer good" but "did the agent take a sensible path". createTrajectoryScorer scores the tool-call sequence against four dimensions — an ordered subsequence it should contain, tools that must appear in any order, tools that must not appear at all, and a call budget. Exceeding the budget or hitting a forbidden tool degrades the score proportionally rather than zeroing it.

It reads the same decision sequence the regression diff uses (buildDecisionSequence), so "a decision point" means one thing across the whole system rather than two similar things in two packages.

import { createTrajectoryScorer, trajectoryScorerFor, contentSimilarity } from '@gnldev/evals';

const path = createTrajectoryScorer({
  expectedTools: ['searchDocs', 'summarise'],  // ordered subsequence; extras in between are fine
  requiredTools: ['citeSource'],               // must appear, order irrelevant
  forbiddenTools: ['deleteRecord'],            // a hit degrades the score proportionally
  maxToolCalls: 8,                             // over budget degrades, it does not zero
});

// bound to a reader: the sample carries a runId and the scorer looks the run up
const fromJournal = trajectoryScorerFor(reader, { requiredTools: ['citeSource'] });

await evalDataset({ dataset, run, scorers: [path, contentSimilarity()], journal });