Retry & Fallback

Also known as: Retry with backoff, Graceful degradation

Retry & fallback re-attempts a failed step under a declared policy of attempts and backoff, and switches to an alternative path when retries are exhausted or the error is not retryable.

01Problem it solves

Remote calls fail transiently (timeouts, rate limits, overloaded providers). Failing the whole workflow on the first error is fragile, while unbounded retries amplify outages.

02Use when / Avoid when

Use when

Steps call networks, APIs or models with transient failure modes.
An acceptable alternative exists: a second provider, a cached value or a reduced answer.
Retry behaviour should be declared per step rather than written ad hoc.

Avoid when

The step is not idempotent and a duplicate would cause harm.
The failure is deterministic (validation error, bad input); retrying only wastes time.
Callers need a fast failure more than an eventual success.

03How it works

1AttemptThe step runs with a timeout.
2ClassifyOn error, decide whether it is retryable.
3Back offWait with exponential backoff and jitter, then retry up to the maximum attempts.
4Fall backIf retries are exhausted, run the fallback path or fail with a clear error.

04Capabilities

Capability	What it means
Automatic retries	Failed steps are re-attempted according to a declared policy (attempts, backoff, retryable errors) rather than ad hoc code.
Fallback paths	When a step keeps failing, execution switches to a declared alternative (another provider, model, cache or degraded answer).

05Tradeoffs

Aspect	You gain	You pay
Availability	Absorbs transient failures without human action.	Adds latency on the failure path.
Load	Backoff spreads recovery traffic.	Retries at several layers multiply load during an outage.
Quality	Fallbacks keep the system answering.	Fallback answers may be degraded; users may not notice.

06Failure considerations

Failure mode	Mitigation
Retry storms: every layer retries and multiplies traffic on a struggling dependency.	Retry at one layer only, add jitter and pair with a circuit breaker.
Duplicate side effects from retried non-idempotent calls.	Use idempotency keys or make the operation naturally idempotent.
Fallback hides a persistent outage.	Emit metrics and alerts on fallback usage.

07Implementations

Examples of products and frameworks that implement this pattern. Listed as evidence, not endorsement.

Implementation	Mechanism
TemporalDurable execution engine	Declarative retry policies per Activity (attempts, backoff, non-retryable errors).
Apache AirflowWorkflow scheduler	Per-task retries and retry delays; trigger rules route around failures.
PrefectWorkflow scheduler	Retries with configurable delays on tasks and flows.
AWS Step FunctionsCloud workflow service	Retry and Catch fields on each state.
Azure Durable FunctionsCloud workflow service	Activity calls accept retry options.
DagsterWorkflow scheduler	Retry policies on ops and assets.

09Requirements that lead here

I need retries that survive process restarts. Primary