Reliability · Established
Checkpointing
Also known as: Snapshotting, State persistence
Checkpointing saves a snapshot of execution state at defined points, so that after a failure, pause or inspection the run can be restored from the latest snapshot rather than restarted from the beginning.
01Problem it solves
Long or expensive runs (multi-step agents, training jobs, stream processing) lose all work when interrupted, and cannot be paused for review or rewound to try a different path.
02Use when / Avoid when
Use when
- Each step is expensive enough that redoing completed work is costly.
- You want to pause for human review and resume later, possibly in another process.
- You need to inspect, replay or branch from earlier states (time travel, debugging).
- A full durable execution engine is more than the problem needs.
Avoid when
- Side effects between checkpoints cannot tolerate being repeated after restore.
- State is too large to snapshot at the required frequency.
- Steps are cheap and a full restart is simpler.
03How it works
- 1Run stepExecute a step of the process.
- 2SnapshotSerialise the relevant state and write it to a checkpoint store, keyed by run and step.
- 3InterruptThe run crashes, is paused, or is stopped for inspection.
- 4RestoreLoad the latest (or a chosen) checkpoint.
- 5ContinueResume execution from the step after that checkpoint.
04Capabilities
| Capability | What it means |
|---|---|
| Persistent execution state | Progress, variables and position in the workflow are stored outside process memory, so a crash does not lose where execution was. |
| Resume after failure | A stopped or crashed run continues from its last recorded point instead of starting over. |
| Wait for external input | Execution pauses until an outside signal arrives (an approval, a callback, a human edit) without holding compute while it waits. |
05Tradeoffs
| Aspect | You gain | You pay |
|---|---|---|
| Recovery | Bounded rework after failure: at most one interval is lost. | Checkpoint frequency trades write overhead against lost work. |
| Flexibility | Enables pause, resume, replay and branching of runs. | State must be serialisable and schema changes must stay compatible. |
06Failure considerations
| Failure mode | Mitigation |
|---|---|
| Side effects between the last checkpoint and the crash are repeated on restore. | Checkpoint after side effects and make those effects idempotent. |
| Checkpoint schema changes make old snapshots unreadable. | Version the state schema and migrate on load. |
| The checkpoint store fills up. | Apply retention per run and keep only the checkpoints you need to restore or audit. |
07Implementations
Examples of products and frameworks that implement this pattern. Listed as evidence, not endorsement.
| Implementation | Mechanism |
|---|---|
| LangGraphAgent framework | Checkpointers persist graph state per thread after each step. |
| PrefectWorkflow scheduler | Result persistence and caching let reruns skip tasks that already completed. |