Databricks Architecture Diagram: Lakehouse, Clusters & Unity Catalog (2026)
How to create Databricks architecture diagrams. Covers the control plane/data plane split, Unity Catalog governance, Delta Lake, cluster types, SQL warehouses, and how Databricks compares to Snowflake — with AI prompt templates.
R
Ryan·Senior AI Engineer
Databricks architecture diagrams trip people up for one recurring reason: Databricks splits itself into a control plane it manages and a data plane that runs inside your own cloud account (AWS, Azure, or GCP), and a diagram that doesn't show that boundary hides where your data actually lives and which network you need to secure. This guide covers the components worth putting on a Databricks lakehouse diagram, from Unity Catalog governance down to cluster types.
Control plane vs data plane
- Control plane (Databricks-managed): The web UI, job scheduler, cluster manager, notebook backend, and REST APIs. This runs in Databricks' own AWS/Azure/GCP account and never touches your raw data directly — it orchestrates.
- Data plane (customer-managed): The actual compute clusters and the object storage they read/write, running inside your cloud account and VPC. Your data never leaves your cloud boundary unless you explicitly move it. Serverless compute is the exception — Databricks-managed compute that still keeps data access scoped to your storage.
- Why this matters on a diagram: Security and compliance reviewers care most about this line. Draw it as an explicit boundary — control plane on one side, your VPC/subnet with clusters and storage on the other — rather than one undifferentiated “Databricks” box.
Core components of a Databricks architecture diagram
- Workspace: The top-level container for notebooks, jobs, and dashboards, tied to one metastore and one cloud region.
- Unity Catalog: The unified governance layer — a metastore containing catalogs, schemas, and tables/volumes, with fine-grained access control, lineage tracking, and audit logging that spans every workspace attached to it. Show it as sitting above individual workspaces since one metastore typically serves many.
- Delta Lake tables: The open storage format (Parquet files plus a transaction log) underlying most Databricks tables, providing ACID transactions, time travel, and schema enforcement on top of cloud object storage (S3, ADLS, GCS).
- Cluster types: All-purpose clusters (interactive notebook work), job clusters (spun up per scheduled job, torn down after), SQL warehouses (for BI/SQL workloads), and serverless compute (Databricks-managed, no cluster configuration). Each has a different cost and isolation profile worth annotating.
- Lakeflow / Delta Live Tables: Declarative pipeline definitions for building and maintaining Delta tables with automatic dependency management, data-quality expectations, and incremental processing — Databricks' answer to hand-rolled ETL DAGs.
- MLflow: Experiment tracking, model registry, and model serving, integrated with Unity Catalog for model governance alongside data governance.
Cluster and compute types compared
| Compute type | Lifecycle | Best for |
|---|---|---|
| All-purpose cluster | Manually started/stopped, shared | Interactive notebook development, exploration |
| Job cluster | Created per job run, terminated after | Scheduled production pipelines — isolated, cost-predictable |
| SQL warehouse | Auto-scaling, auto-suspend | BI dashboards and ad hoc SQL against Delta tables |
| Serverless compute | Databricks-managed, no cluster config | Fast-start notebooks and jobs without capacity planning |
Prompt examples for Databricks architecture diagrams
Medallion architecture on Databricks with Unity Catalog
"Databricks lakehouse using the medallion architecture on AWS. Raw data lands in an S3 bucket (Bronze layer, Delta format) via Auto Loader from Kinesis and batch S3 file drops. A Lakeflow (Delta Live Tables) pipeline cleans and deduplicates data into a Silver layer, then aggregates into business-level Gold tables. All three layers registered in Unity Catalog under a 'prod' catalog with schema-level access control (analysts read Gold only, data engineers read/write Bronze and Silver). BI users query Gold tables via a SQL warehouse connected to Tableau. Show the control plane (job scheduler, Unity Catalog metastore) as a Databricks-managed box, and the S3 buckets plus job clusters as customer-managed data plane inside a VPC."
MLOps pipeline with MLflow and model serving
"MLOps architecture on Databricks. Feature engineering job (job cluster) reads Gold Delta tables from Unity Catalog and writes to a Feature Store table. Training job (GPU job cluster) pulls features, trains a model, and logs metrics/artifacts to MLflow Tracking. Best model registered in the Unity Catalog Model Registry, promoted from 'Staging' to 'Production' alias after evaluation. Production model deployed to a Databricks Model Serving endpoint (serverless, auto-scaling) that a downstream application calls via REST for real-time inference. Show the training loop (feature store → training job → MLflow tracking → model registry) as one flow, and the serving path (model registry → serving endpoint → application) as a separate flow branching off the registry."
Cross-cloud Unity Catalog governance
"Databricks governance architecture with a single Unity Catalog metastore shared across two workspaces — one on AWS (us-east-1) for data engineering, one on Azure (East US) for a regional ML team. Metastore holds catalogs 'raw', 'curated', and 'ml_features', each with row-level and column-level security policies (mask PII columns for non-privileged roles). Both workspaces reference the same metastore for consistent access policy, lineage, and audit logs, while each workspace's compute stays in its own cloud account. Show the shared Unity Catalog metastore as a central governance node with two workspace boxes (AWS, Azure) each pointing to it, and annotate that data access policies are enforced identically regardless of which workspace a query originates from."
Databricks vs Snowflake, at a glance
Both are lakehouse-capable platforms in 2026, but they start from opposite ends: Databricks began as a Spark-native compute platform and grew SQL/warehouse features on top, while Snowflake began as a SQL warehouse and grew Spark-like open-table support on top. If you're diagramming a stack that's primarily notebook-driven ML/ETL with Python and Spark, lean Databricks; if it's primarily SQL-first BI and governed sharing, lean Snowflake. See the Snowflake architecture diagram guide for the warehouse-first comparison point.
What to annotate on a Databricks diagram
- Control plane / data plane boundary: Which cloud account and VPC hold your actual data and compute.
- Cluster type per workload: All-purpose vs job vs SQL warehouse vs serverless, since each has a different cost model.
- Unity Catalog scope: Which catalogs/schemas a metastore governs and which workspaces attach to it.
- Storage format and location: Delta Lake on S3/ADLS/GCS, plus which layer (Bronze/Silver/Gold) each table belongs to.
- Access control model: Row/column-level security policies and which roles they apply to.
Related guides: data lakehouse architecture, Snowflake architecture diagrams, modern data stack architecture, and dbt architecture diagrams.
Ready to try it yourself?
Start Creating - Free