Semantic duplicate-candidate gate

It never silently skips, blocks or tells the model "already done" on your behalf; the final word is always deterministic field equality plus a human. It is best-effort and fail-open: if your embedder is unreachable or no candidate clears the bar, behavior is today's behavior — no regression, and no guarantee either.

The decision hierarchy: deterministic > human gate > probabilistic. This layer is the third class, serving the first two as a candidate finder. It does not replace the hash/claim/confirm/critical layers; it runs beneath them, and because its only exit is the approval question it refuses to start where no approvals channel exists — action: 'suspend' and scope: 'thread' are config-time requirements.

We measured where these rungs earn their keep, and it was not where we first assumed. A competent tool model canonicalises for you: "one more of that television" comes back as TV-42, byte-identical to the earlier call, so the exact-hash layer takes it and the semantic rungs never get their turn. This layer earns its keep where the model passes text through as written — support-ticket subjects, customer and company names, free-text descriptions.

Both declarations are required; with either one absent the layer is entirely inert: the run-level limits.sideEffectDuplicates.semantic block, and the tool-level semanticIdentity declaration. A tool that does not declare its identity never passes through this layer at all — and leaving keys empty is a config-time throw, because a gate that looks installed and does nothing is precisely the false confidence it exists to prevent.

The order is fixed, and each rung takes over the residue the previous one left. None of them can conclude anything on its own:

The layer lives in @gnldev/durable; you own the embed closure, the provider and the bill:

import { createGnl, gnlTool, type RunLimits } from '@gnldev/durable';

Start with the run block and the tool declaration. embedModelId is a required stamp: vectors from different models are apples and oranges, and a record whose stamp differs is excluded from the comparison rather than silently compared.

// Double opt-in: the run-level block AND the tool-level declaration — either absent, layer inert.
const limits: RunLimits = {
  sideEffectDuplicates: {
    action: 'suspend', scope: 'thread',            // required — config-time throw otherwise
    semantic: {
      embed: myEmbed,                              // (texts: string[]) => Promise<number[][]>
      embedModelId: 'text-embedding-3-small',      // required stamp — mixed-model cosine is meaningless
      minSimilarity: 0.6,                          // candidate threshold (recall side; misses are safe)
    },
  },
};

const createProduct = gnlTool(tool({ /* … */ }), {
  sideEffect: true,
  semanticIdentity: {
    keys: ['sku'],                                 // the business identity — REQUIRED, non-empty
    amountFields: ['price'],                       // identity-equal + amounts differ → its own question
    discriminatorFields: ['cancel'],               // negation gate: differ → deterministically not a duplicate
    describe: (args: any) => `create product ${args.sku}`, // the PII boundary: ONLY this reaches the embedder
  },
});

For the critical profile the recommended setup is a local embedder: it removes the API bill, the rate limit and the PII question in one move. Stamp the quantization too — a re-quantized model produces different vectors. The same closure also serves @gnldev/memory's semantic recall.

// npm i @huggingface/transformers   (~150-400MB RAM at runtime, ~5-20ms per short sentence on CPU)
import { pipeline } from '@huggingface/transformers';

const extractor = await pipeline('feature-extraction', 'Xenova/multilingual-e5-small', { dtype: 'q8' });
const myEmbed = async (texts: string[]) => {
  // e5 family quirk: inputs want a "query: " prefix — bake it into the adapter, never into callers.
  const out = await extractor(texts.map((s) => `query: ${s}`), { pooling: 'mean', normalize: true });
  return out.tolist();
};
// embedModelId: 'local:multilingual-e5-small@q8'  ← stamp the QUANTIZATION too: a re-quantized model
// produces different vectors, and the stamp is what keeps old records out of the comparison.

Records live in YOUR journal under the thread prefix (~3KB each; no vector database, no index) and die with the thread in the same purgeThread sweep as everything else. Nothing is written for failed or suspended work. Cost model: ~1 embed call per guarded side-effect call on the happy path.

The certificate is not ceremony. On identical fixtures with the identical prompt we measured it: one model answered 43% of the paraphrase pairs correctly and another 100%. A third, from a family unrelated to the one that wrote the fixtures, scored 93% — so the spread is about the model, not about whose phrasing it recognises.

That is why a judge you have not measured is a layer that looks installed and is not. A missing, weak (recall below 0.70 or false alarms above 0.05), model-mismatched or prompt-version-mismatched certificate is a config-time throw — the sibling of v1's empty-keys throw. Swapping the model or upgrading past a prompt-version bump invalidates it and the exam must be re-sat.

semantic: {
  embed: myEmbed, embedModelId: 'local:multilingual-e5-small@q8',
  rules: true,                                  // the deterministic ladder — the whole configuration
  judge: {
    // TRANSPORT ONLY: the framework renders the prompt, you own the model and the bill.
    complete: async ({ system, user }) => (await myModel(system, user)).text,
    judgeModelId: 'your-judge-model',
    qualification: cert,                        // from @gnldev/semantic-qualify — REQUIRED
    maxCallsPerRun: 10,                         // journal-backed slots; survive resume
    timeoutMs: 8000,
  },
},

npx gnl-semantic-qualify --judge ./my-judge.mjs --model your-judge-model   # → gnl-judge-cert.json

The bench evaluates blind: opaque ids, shuffled order, labels never sent to the model, and the bars read from the same exported constants the runtime enforces. Point it at your own fixtures with --fixtures; the published set stops being held out the moment it is published. And the certificate stamps EXAM performance, not field accuracy.

Count first, judge second. Studio's semantic card reports scan.grayCalls: the number of CALLS that produced a gray residue — the price quote for what a judge would cost you. On measured traffic the judge spoke 12 times (across 126 tool-calling turns) and none of them produced a question; the gray band was genuinely made of different work. Run with rules on and judge off for a while, read the counter, then decide.

All five are deliberate boundaries, and none of them was left to be discovered later: