Eval gate (promote governance)

With Agent versioning, an agent can have several versions kept in the journal, and whichever version you choose is activated with promote. But if the decision to "ship the new system prompt/model to production" is left to human discretion, there's a regression risk. evalGate automates this: when createStudioApp is given a { datasetId, minAvg }, every promote request FIRST runs that eval dataset suite; if not all of the suite's aggregate scores clear the minAvg threshold, promote is rejected with 412 — the version doesn't go active.

Typical scenario: you define a regression dataset for your production agent (e.g. frequently asked questions + expected answers); when an engineer adds a new system prompt version and tries to promote it, Studio automatically blocks it if the suite doesn't pass — the decision itself (passed/failed + aggregate scores) is recorded in the audit log as an agent.gate event, leaving a permanent trail of who tried what and when.

evalGate isn't a separate package — it's an option passed to @gnldev/studio's createStudioApp (or createStudioApi) call. For it to activate, you MUST supply the same call with a datasets (StudioDatasets contract) — without datasets, a promote request returns 501.

import { createStudioApp } from '@gnldev/studio';
import type { StudioDatasets, EvalDatasetResultLike } from '@gnldev/studio';
import { evalDataset } from '@gnldev/evals';

1) Implement the StudioDatasets contract — the Studio core doesn't depend on @gnldev/evals; inside datasets.run(id) you call evalDataset yourself and return a result that's structurally compatible with EvalDatasetResultLike:

const datasets: StudioDatasets = {
  list: () => [{ id: 'regression', cases: 12, description: 'Frequently asked questions' }],
  run: async (id) => {
    const report = await evalDataset({
      dataset: regressionDataset, // { id: 'regression', cases: [...] }
      run: async (input, { runId }) => gnl.run('starwars', { runId, prompt: input }),
      scorers: [exactMatch()],
      journal,
    });
    return report; // { datasetId, cases, aggregate }
  },
};

2) Give evalGate to createStudioApp — datasetId determines which suite runs, minAvg (default 0.5) sets the passing threshold:

app.route('/studio', createStudioApp({
  reader: toJournal(storage.runs),
  gnl: createStudioRunner(gnl, config, { toJsonSchema: aiToolSchema }),
  datasets,                                     // required — without it evalGate returns 501
  evalGate: { datasetId: 'regression', minAvg: 0.7 },
  auth,
}));

3) The promote request now goes through the gate — when POST /managed-agents/:name/promote is called, Studio first runs datasets.run('regression'); if any aggregate score is below 0.7 (`failing`) the request is rejected with 412, otherwise the version is activated:

POST /studio/api/managed-agents/starwars/promote
{ "version": 3 }

// 412 Precondition Failed
{
  "error": "the eval gate FAILED: exact-match=0.42<0.7 — promotion refused",
  "aggregate": { "exact-match": 0.42 }
}
// an 'agent.gate' event lands in the audit log: { version: 3, datasetId: 'regression', minAvg: 0.7, aggregate, passed: false }

POST /studio/api/managed-agents/starwars/promote
{ "version": 3 }

// 200 OK
{ "ok": true, "name": "starwars", "active": 3, "previous": 2 }
// 'agent.gate' (passed: true) lands in the audit log first, then 'agent.promote'