Cross-run cache

Reuse within a run is already solved: that is what replay does. What replay cannot do is share work between two different runs — and that is where the repeated cost usually is. Two runs asking the same RAG question embed the same text twice; two runs summarising the same document pay for it twice.

The key is run-independent (<namespace>:<hash>), so a result computed under one runId is found under another. That is the whole idea: not a faster journal, but the gap the journal was never meant to cover.

getOrCompute is the one you will reach for: it returns the cached value if there is one, and otherwise computes, stores and returns it. get and set are there when you want the two halves separately, and ttlMs applies when the storage backend supports the TTL capability.

import { createCache } from '@gnldev/cache';

const cache = createCache(storage.cache, 'embeddings', {
  onError: (op, err) => logger.warn({ op, err }, 'cache degraded'),
});

// run A computes it; run B finds it — the key does not carry a runId
const vector = await cache.getOrCompute(
  { text: question },
  () => embed(question),
  { ttlMs: 86_400_000 },
);

A cache is an accelerator, not a source of truth, so a store failure must not fail the run. Every operation degrades instead of throwing: a failed get becomes a miss, a failed set becomes a no-op, and getOrCompute still computes and still returns the right answer. Errors are reported through the onError callback rather than swallowed silently.

This is deliberately the opposite of how the run journal and memory behave, and the difference is worth stating: those are strict, because losing a journal write would lose the guarantee. Losing a cache write only costs time.

One small design detail that saves a real bug: the stored value is wrapped as { v }. Without the wrapper, a cached undefined and "nothing is cached here" would be indistinguishable, and every lookup of a legitimately-undefined result would recompute.

stats() returns hit and miss counters, the hit rate and a key count. These are in-process counters: they reset when the process restarts, because the store's job is storing values, not counting them.

invalidate(key) deletes that one key. invalidate() with no argument deletes only the keys this instance has seen — the CacheStore port offers no key enumeration, so there is no way to clear a store wholesale from here. It is not a flush, and the page says so because the name invites the assumption.

cache.stats();
// { hits: 41, misses: 9, hitRate: 0.82, size: 50 }

await cache.invalidate({ text: question });   // one key
await cache.invalidate();                     // only what THIS instance has seen