DAG

Also known as: Directed acyclic graph, Task graph, Pipeline

A DAG orchestrates tasks as a directed acyclic graph of dependencies: each task runs once its upstream tasks have succeeded, and independent branches can run in parallel.

01Problem it solves

Multi-step data and batch processes break when steps run in the wrong order or when a failure in the middle forces the whole pipeline to rerun from scratch.

02Use when / Avoid when

Use when

Tasks have fixed, known dependencies and no loops.
Runs are scheduled or triggered in batches (ETL, ML training, reports).
You want to rerun only failed tasks and their downstream dependents.
Lineage and a visual overview of the pipeline matter.

Avoid when

The flow needs loops, long waits or decisions that revisit earlier steps; use a state machine or durable execution.
The structure is only known at runtime per request (dynamic agent planning).
Latency per request matters more than batch throughput.

03How it works

1DefineDeclare tasks and their upstream dependencies; the scheduler verifies there are no cycles.
2OrderTasks are ordered topologically.
3ExecuteTasks whose dependencies are met run, independent branches in parallel.
4Handle failureA failed task is retried; if it still fails, downstream tasks are skipped or blocked.
5ResumeA rerun can start from the failed task, reusing successful upstream results.

04Capabilities

Capability	What it means
Dependency ordering	Steps start only when the steps they depend on have finished, as declared in an explicit dependency graph.
Parallel execution	Independent units of work run concurrently instead of one after another.
Automatic retries	Failed steps are re-attempted according to a declared policy (attempts, backoff, retryable errors) rather than ad hoc code.

05Tradeoffs

Aspect	You gain	You pay
Predictability	Static structure is easy to schedule, visualise and audit.	Dynamic control flow is awkward or limited.
Operations	Partial reruns and backfills are routine.	Scheduler-centric model; not built for low-latency, per-request work.

06Failure considerations

Failure mode	Mitigation
Non-idempotent tasks produce duplicates on rerun or backfill.	Write outputs by partition key and overwrite rather than append.
A hidden dependency (shared table or file) is missing from the graph.	Declare data dependencies explicitly or use data-aware scheduling.
Large fan-out tasks overload the scheduler.	Batch dynamic tasks and limit concurrency per pool.

07Implementations

Examples of products and frameworks that implement this pattern. Listed as evidence, not endorsement.

Implementation	Mechanism
Apache AirflowWorkflow scheduler	DAGs of tasks with explicit upstream/downstream dependencies.
DagsterWorkflow scheduler	Asset and op graphs with explicit dependencies.

09Requirements that lead here

Several workers must execute in parallel and rejoin. Also consider