The problem with agentic pipelines and crashes
A single-agent workflow that completes in 30 seconds doesn't need durable execution. If it crashes, you restart it.
Multi-agent pipelines that run for hours — processing a batch of documents, running overnight compliance checks, training data collection — need something different. When a node fails at step 7 of 20, you don't want to restart from step 1. You want to resume from step 7.
The naive approach is to checkpoint after every step. Write state to disk, reload on crash, continue. This works until you have to handle:
- Re-execution semantics. If step 7 was partially completed when the crash happened — the LLM call returned but the result wasn't persisted — do you re-run it? What if the result was non-deterministic?
- Partial writes. Writing state to SQLite is not atomic with running the LLM call. A crash between the LLM response and the write leaves you with ambiguous state.
- Concurrency. Parallel crews checkpoint concurrent branches. Merging concurrent checkpoint state is a distributed transactions problem.
Why we didn't use event sourcing
Event sourcing — storing every state transition as an append-only event log — is the conceptually clean solution. Replay the events to reconstruct state. Handle re-execution by replaying from the last committed event.
We evaluated this approach and rejected it for two reasons:
LLM calls are not idempotent by default. Replaying an event log that includes LLM calls requires either storing the LLM output in the event log (which makes the log large) or re-running the LLM call (which may produce a different result). Neither option is clean.
The replay boundary is hard to define. At what point is a "step" committed? When the LLM returns? When the result is validated? When downstream steps have consumed it? Event sourcing requires a precise definition of causality that gets complicated fast with parallel execution.
What we built instead: write-ahead logging with idempotency keys
The DurableWorkflowExecutor uses write-ahead logging (WAL) with explicit idempotency keys.
Before each step executes: 1. Write a `STARTED` record to the WAL: `{ step_id, idempotency_key, timestamp }` 2. Run the step (LLM call, tool call, etc.) 3. Write a `COMMITTED` record to the WAL: `{ step_id, output_hash, timestamp }`
On resume after crash: 1. Read the WAL from the last `COMMITTED` record 2. For any step with a `STARTED` but no `COMMITTED`: re-run with the same idempotency key 3. For any step with a `COMMITTED`: skip re-execution, load the persisted output
The idempotency key is a SHA-256 of `(run_id, step_index, input_hash)`. For deterministic steps, the same input produces the same output — re-execution is safe. For non-deterministic steps (most LLM calls), re-execution produces a new output, and we persist the new output under the same key.
The checkpoint schema
CREATE TABLE checkpoints (
run_id TEXT NOT NULL,
step_id TEXT NOT NULL,
step_index INTEGER NOT NULL,
status TEXT NOT NULL, -- STARTED | COMMITTED | SKIPPED
input_hash TEXT,
output_hash TEXT,
output_data BLOB, -- compressed JSON
started_at TEXT,
committed_at TEXT,
idempotency_key TEXT NOT NULL,
PRIMARY KEY (run_id, step_id)
);The `output_data` column stores the compressed step output. On resume, this is decompressed and injected as the step's "previous result" without re-running the LLM.
Parallel execution and checkpointing
For parallel crews, each branch checkpoints independently. The merge step waits for all branches to reach `COMMITTED` status before proceeding.
executor = DurableWorkflowExecutor( db_path="meshflow_runs.db", backend="sqlite", # or "postgres" or "s3" )
result = await executor.run( wf, task="Process Q3 compliance report", resume_run_id="run_abc123", # resumes from last checkpoint ) ```
If `resume_run_id` is provided, the executor loads the WAL from the database and skips any step with a `COMMITTED` record. The workflow picks up from the last uncommitted step.
Backends
We support three checkpoint backends: - SQLite (default): zero-ops, single-process, good for development and small deployments - PostgreSQL: multi-process, ACID guarantees, suitable for production - S3 + DynamoDB: serverless, infinitely scalable, suitable for batch jobs
The backend is swappable with one configuration change. The WAL schema is identical across backends.
What we learned
The hardest part of building durable execution isn't the happy path — it's the edge cases:
- Zombie steps. A step writes `STARTED` but the process dies before the LLM call even goes out. On resume, should you re-run it? (Yes, always.)
- Timeout-then-succeed. The HTTP request to the LLM API times out from the client's perspective, but the server processed it and the response is in flight. On resume, you re-run, and the LLM returns a different response. (We accept this — non-determinism is a property of the system.)
- Concurrent resumes. Two processes both try to resume the same run. (The WAL is append-only; the second writer detects the conflict via the idempotency key and yields.)
Durable execution is a solved problem in distributed systems. We chose the simplest approach that handles the real failure modes rather than building a full saga coordinator.