Apache Airflow Architecture Diagram: Schedulers, Executors & DAGs Explained (2026)
How to create Apache Airflow architecture diagrams. Covers the scheduler, webserver, metadata database, DAG processor, executor types (Celery, Kubernetes, Local), and managed platforms like MWAA and Cloud Composer — with AI prompt templates.
R
Ryan·Senior AI Engineer
Apache Airflow architecture diagrams are a recurring ask from data engineering teams because Airflow's component split — scheduler, webserver, metadata database, and executor — isn't obvious from the DAG code alone, and misdiagramming it usually means confusing “where a task is defined” with “where a task actually runs.” Airflow orchestrates workflows as Directed Acyclic Graphs (DAGs) of tasks, but the runtime that parses, schedules, and executes those DAGs is a distributed system with several moving parts. This guide covers the components worth putting on an Airflow diagram and how they connect.
Core components of an Airflow architecture diagram
- Scheduler: The always-on process that monitors DAGs, evaluates task dependencies and schedules, and triggers task instances when they're ready to run. Show this as the central orchestration node — nothing runs without the scheduler deciding it should.
- DAG processor: Airflow 2.3+ separates DAG file parsing into its own process (or dedicated deployment in Airflow 3.0) so that parsing untrusted or slow DAG code can't block the scheduler's core loop. Draw it as a distinct box feeding parsed DAG definitions to the scheduler and metadata database.
- Metadata database: Postgres or MySQL storing DAG state, task instance history, connections, variables, and XComs (cross-task communication data). Every other component reads from or writes to this database, which makes it the system's central dependency and usual bottleneck at scale.
- Webserver / UI: The Flask-based web UI for monitoring DAG runs, viewing logs, and manually triggering or clearing tasks. It reads from the metadata database and is stateless, so it's straightforward to run multiple replicas behind a load balancer.
- Triggerer: A dedicated process for deferrable (“async-aware”) operators — tasks that poll for an external condition (a file landing, a job finishing) without occupying a worker slot while waiting. Show it as a separate lightweight process alongside the scheduler for any diagram involving sensors or long-poll tasks.
- Executor: The component that actually runs task instances, ranging from in-process (Local) to distributed (Celery, Kubernetes). This is the piece most worth zooming in on since it determines your scaling and isolation model.
- DAG files (bag of DAGs): Python files defining workflows, typically synced from a Git repo via a sidecar, shared volume, or object-storage sync (S3, GCS) into the scheduler, DAG processor, and workers.
Executor types compared
| Executor | Execution model | Best for |
|---|---|---|
| Local | Subprocesses on the scheduler host | Development, single-node deployments, small pipelines |
| Celery | Distributed worker pool via a message broker (Redis/RabbitMQ) | High task volume with a stable, always-on worker fleet |
| Kubernetes | One pod per task instance, spun up on demand | Per-task resource isolation, dependency conflicts between DAGs, scale-to-zero |
| CeleryKubernetes | Hybrid — routes tasks to Celery or Kubernetes per-task | Mixed workloads: fast lightweight tasks on Celery, heavy isolated tasks on Kubernetes pods |
Managed and self-hosted deployment options
- Amazon MWAA (Managed Workflows for Apache Airflow): AWS-managed scheduler, workers, and webserver with DAGs synced from S3, IAM-based access, and VPC networking — no metadata database or Kubernetes cluster to operate yourself.
- Google Cloud Composer: Airflow on a managed GKE cluster with Cloud SQL as the metadata database, integrated with GCP IAM and Cloud Logging.
- Astronomer: A managed Airflow platform (Astro) plus enterprise features like DAG-level alerting, lineage, and a hybrid deployment model for teams that want managed control planes with self-hosted data planes.
- Self-hosted on Kubernetes: The official Airflow Helm chart deploys scheduler, webserver, triggerer, and (optionally) a Postgres/Redis dependency stack as separate Kubernetes deployments — the most common self-hosted pattern for teams already on K8s.
Prompt examples for Airflow architecture diagrams
Self-hosted Airflow on Kubernetes with CeleryExecutor
"Apache Airflow architecture on Kubernetes using the official Helm chart with CeleryExecutor. Components: Scheduler pod, DAG Processor pod, Webserver pod (2 replicas behind a Service/Ingress), Triggerer pod, Redis (Celery broker), Postgres (metadata database, RDS-managed), and a Celery worker Deployment (autoscaled 2-10 pods via KEDA based on queue depth). DAG files synced from a Git repo into all pods via a git-sync sidecar on a shared volume, refreshed every 60 seconds. Show the scheduler and DAG processor both reading DAG files and writing to Postgres, workers pulling tasks from Redis and writing task state back to Postgres, and the webserver reading from Postgres only. Annotate KEDA autoscaling on the worker Deployment."
MWAA data pipeline triggering dbt and Snowflake
"Data pipeline orchestrated by Amazon MWAA. DAGs stored in an S3 bucket synced to MWAA's managed environment. Daily DAG: extract task pulls from a Postgres OLTP database via a PythonOperator, loads raw data into S3 (Parquet), then a dbt Cloud operator triggers a dbt job that transforms and loads into Snowflake, followed by a data-quality check task (Great Expectations) and a Slack notification task on failure. Show MWAA's managed scheduler/webserver/worker environment as one box with IAM role annotations, and the DAG's task dependency chain (extract → load → dbt transform → quality check → notify) as a separate flow diagram beneath it."
Multi-tenant Airflow with per-team KubernetesExecutor isolation
"Multi-tenant Airflow deployment using KubernetesExecutor for per-task isolation across three teams (data-eng, ml, analytics). Single scheduler and metadata database shared across tenants, but each task instance runs as its own Kubernetes pod using a team-specific namespace and resource-limited pod template (CPU/memory requests set per team's DAG defaults) to prevent one team's heavy job from starving another's. Show the scheduler dispatching pod-creation requests to the Kubernetes API server, with three namespaces (data-eng, ml, analytics) each receiving ephemeral task pods, and a shared Postgres metadata database and S3-backed DAG bag common to all three."
What to annotate on an Airflow diagram
- Executor type: Local, Celery, Kubernetes, or CeleryKubernetes — it determines where task code actually runs and what isolation guarantees exist between tasks.
- DAG sync mechanism: Git-sync sidecar, S3/GCS bundle, or baked into the container image — this affects deploy latency and whether a bad DAG can be rolled back quickly.
- Metadata database sizing: Connection pool limits and whether PgBouncer sits in front of Postgres, since the scheduler and every worker hit this database constantly.
- Concurrency limits:
parallelism,dag_concurrency, and pool slots — the knobs that prevent one DAG from starving others of worker capacity. - Downstream systems: What each task actually touches (warehouses, APIs, object storage) — Airflow itself is an orchestrator, so the diagram's value is showing what it orchestrates, not just its internals.
Related guides: Helm chart architecture diagrams, dbt architecture diagrams, modern data stack architecture, and data mesh architecture.
Ready to try it yourself?
Start Creating - Free