What happens when your agent crashes?
GNL teamdurabilitycrash-recoveryside-effectsapprovals
The work starts over, an email goes out twice, an approval is lost. Each has a fix, and one moment has no fix at all — only an honest answer.
The short answer#
In most agents, a crash mid-run throws the run away. The retry starts from the top. The model plans again, the tools run again, and anything that left the process — an email, a ticket, a payment — can happen a second time.
With GNL you call again with the same runId. Finished steps come back from a record, only the unfinished one runs, and a tool that changes the outside world runs at most once. When GNL cannot know whether it ran, it does not guess: it stops and asks a person.
What breaks#
Take a support agent behind a queue. For each message it looks up the customer, opens a ticket and emails them. A deploy, a timeout or an out-of-memory kill can stop it between any two of those steps.
import { generateText, stepCountIs } from 'ai';
// A queue worker runs this for each incoming message.
await generateText({ model, tools, stopWhen: stepCountIs(10), prompt: job.message });
// The process dies after createTicket, before sendEmail. The retry starts from
// the top, and the model decides again whether to open a ticket.- The work starts over. The retry runs every model step again. You pay for them twice, and the model may take a different path the second time.
- A side effect happens twice. The ticket was opened before the crash. The retry opens another one. The customer gets two emails.
- An approval is lost. A person said yes, the tool ran, and the process died before the answer was saved. The next call sees an approved call with no result and runs it again.
A record of every step#
GNL writes each model step and each tool result to a journal, in your own database, as the run goes. The journal is append-only: it keeps what happened, in order.
Resuming is calling the same function again with the same runId. The steps already in the journal are read back instead of run. The first step that is not there is where the run continues.
import { runDurable, gnlTool } from '@gnldev/durable';
import { PostgresStorage } from '@gnldev/durable/postgres';
const journal = new PostgresStorage({ connectionString: process.env.DATABASE_URL! }).runs;
// Mark the tools that change something outside the agent.
const createTicket = gnlTool(createTicketTool, { sideEffect: true, idempotency: 'args' });
const sendEmail = gnlTool(sendEmailTool, { sideEffect: true, idempotency: 'args' });
// The first attempt and every retry run this same call.
export async function handle(job: { id: string; message: string }) {
return runDurable({
runId: `support:${job.id}`, // the id of THIS job — never a session id
journal,
model,
tools: { lookupCustomer, createTicket, sendEmail },
stopWhen: stepCountIs(10),
prompt: job.message,
});
}The runId names the job, not the conversation. Here it is the queue message's id, so the first attempt and every retry are the same run. Nothing else in the agent changes: the same model, the same tools, the same prompt. See resume a generateText agent after a crash for the full walk-through.
Side effects run once#
Reading back a recorded result is safe for any tool. Running a tool again is not. So you mark the tools that change something outside the agent with sideEffect: true. A marked tool that finished is never run again on resume.
Models also re-plan: the same call can come back under a new call id. idempotency: 'args' says that the same arguments mean the same work, so a re-planned call with the same arguments gets the first call's result. My AI SDK tool ran twice after a retry covers this case.
The one window no one can close#
There is one moment no framework can see through: the tool did its work, and the process died before the result was written down. Did the email go out? From inside the process, there is no way to know.
GNL does not guess here. On resume it stops the run with SideEffectRetryBlockedError and waits for a decision, instead of running the tool again and hoping. When the outside system can answer the question — it accepts an idempotency key, or it can be asked what happened — the tool can say so, and the guarantee reaches all the way to that system.
Approvals survive the crash#
A tool can wait for a person before it runs. With GNL the question and the answer live in the journal, not in a message array you have to save at the right moment.
So the hours between the question and the answer are safe, and so is the moment right after the yes: once the approved call has run, it is recorded, and a crash after it does not run it again. Human approval before an AI SDK tool runs shows the code.
Try it#
If you already have an AI SDK agent, swap generateText for runDurable and give it a journal. That is the whole core. When you want more — a server, Studio to watch and approve runs, workflows, memory — add it a step at a time: from the core to the whole framework.
npm create gnl@latest my-agent, or add the core to an agent you have with npm i @gnldev/durable.