Back to blog

Data Lakehouse Architecture Diagrams: Iceberg, Delta Lake & Unified Storage (2026)

How to create data lakehouse architecture diagrams. Covers Apache Iceberg, Delta Lake, and Apache Hudi table formats, the catalog and compute layers, medallion architecture, and multi-engine query access — with AI prompt templates.

R
Ryan·Senior AI Engineer
·

The data lakehouse is the architecture that ended the long-running tradeoff between data lakes and data warehouses. A data lake gives you cheap, flexible object storage (S3, GCS, ADLS) that can hold any file format, but it lacks ACID transactions, schema enforcement, and the query performance of a warehouse. A data warehouse gives you all of that, but at the cost of proprietary storage, high compute-coupled pricing, and poor support for unstructured or semi-structured data. The lakehouse pattern closes that gap by adding an open table format layer on top of object storage — Apache Iceberg, Delta Lake, or Apache Hudi — that brings warehouse-grade transactions, schema evolution, and time travel directly to files sitting in cheap object storage, queryable by any compute engine that speaks the format.

Because a lakehouse deliberately separates storage, table format, catalog, and compute into independent layers, it is easy to describe in a sentence and genuinely hard to draw correctly. This guide covers what belongs in each layer of a lakehouse architecture diagram, how the three major open table formats differ, the medallion (Bronze/Silver/ Gold) pattern used to organize data inside a lakehouse, and prompt templates for generating accurate diagrams with AI.

What makes a lakehouse a lakehouse

A lakehouse is not a product — it is a pattern built from a specific combination of layers. The defining trait is that the table format metadata layer (not a proprietary storage engine) is what provides ACID guarantees, and that metadata sits directly alongside the data files in the same object storage bucket rather than inside a closed warehouse. Any engine that can read the table format's metadata — Spark, Trino, Snowflake, Databricks, DuckDB — can read and write the same underlying Parquet files consistently, without routing through a single vendor's compute layer. That multi-engine, single-copy property is the core value proposition, and it is exactly what your diagram needs to make visually obvious.

The three open table formats

The table format is the layer that turns a folder of Parquet files into something that behaves like a database table. All three major formats solve the same core problems — atomic commits, schema evolution, snapshot isolation — but differ in origin, governance, and which workload they were originally built to optimize.

Apache Iceberg

Originated at Netflix to solve correctness problems with Hive tables on S3, Iceberg is now governed by the Apache Software Foundation and has the broadest multi-engine support of any table format — Spark, Trino, Flink, Snowflake, Databricks, and DuckDB all read and write Iceberg natively. Iceberg tracks table state through a chain of immutable metadata files and manifest lists, which is what enables reliable snapshot isolation and time travel even under concurrent writers. Iceberg's REST catalog specification (implemented by Snowflake's Polaris Catalog, Databricks' Unity Catalog, Tabular, and others) has become the standard way engines discover and negotiate access to Iceberg tables without depending on a single vendor's proprietary catalog API.

Delta Lake

Originated at Databricks and now governed by the Linux Foundation, Delta Lake is the default table format for Databricks workloads and has deep integration with Spark. Delta tracks changes through a transaction log (the _delta_log directory of JSON and checkpoint files) rather than Iceberg's manifest structure. To reduce lock-in and compete on interoperability, Databricks introduced Delta Lake UniForm, which writes Iceberg- and Hudi-compatible metadata alongside the native Delta log for the same Parquet files — letting an Iceberg-only engine like Snowflake read a Delta table without a conversion job.

Apache Hudi

Originated at Uber, Hudi was purpose-built for high-frequency incremental upserts and streaming ingestion — its record-level indexing makes it the strongest choice for CDC-heavy workloads with very high update volume, such as replicating a large OLTP database into a lake in near real time. Hudi is also an Apache Software Foundation project and supports Spark, Flink, Trino, and Presto.

The 2026 trend across all three formats is convergence rather than competition: Delta UniForm, Iceberg-Delta interop initiatives, and the spread of the Iceberg REST Catalog specification mean the same physical Parquet files are increasingly readable through more than one format's metadata. When you diagram a lakehouse today, it is worth explicitly labeling whether interoperability is in play — it changes which engines can legitimately appear as readers in the diagram.

DimensionApache IcebergDelta LakeApache Hudi
Origin / governanceNetflix → Apache Software FoundationDatabricks → Linux FoundationUber → Apache Software Foundation
Primary strengthBroadest multi-engine supportDeep Spark/Databricks integrationHigh-frequency upserts / CDC
Best-fit use caseMulti-vendor lakehouse queried by many enginesDatabricks-centric pipelines and ML workloadsStreaming CDC replication with very high update volume
Catalog optionsIceberg REST Catalog, Polaris, Glue, Hive Metastore, Unity CatalogUnity Catalog, Hive Metastore, AWS GlueHive Metastore, AWS Glue
Interop pathREST Catalog spec adopted broadlyUniForm writes Iceberg/Hudi-compatible metadataGrowing REST Catalog support

The five layers to diagram

A lakehouse architecture diagram is clearest when it explicitly separates five layers, drawn top to bottom or left to right in the order data flows through them. Collapsing any of these into a single box is the most common mistake — the whole point of the pattern is that each layer is independently swappable.

  • Ingestion layer: Batch sources (files, exports, SaaS APIs pulled via Fivetran/Airbyte) and streaming sources (Kafka, Kinesis) plus CDC pipelines (Debezium reading a database write-ahead log) that continuously land new records into the lake.
  • Storage layer: Object storage — S3, GCS, or Azure Data Lake Storage (ADLS) — holding the actual data files, almost always Parquet (occasionally ORC). This layer is dumb by design: it has no notion of a "table", only files and prefixes.
  • Open table format / metadata layer: Iceberg, Delta, or Hudi metadata — manifest files, snapshots, and schema history — that sits next to the data files and turns them into an ACID-compliant, schema-aware table. This is the layer that makes time travel and schema evolution possible.
  • Catalog layer: Hive Metastore, AWS Glue Catalog, Databricks Unity Catalog, Polaris Catalog, or an Iceberg REST Catalog. Engines query the catalog to resolve a table name to its current metadata location — the catalog is the single shared source of truth that every compute engine consults before touching a table.
  • Compute / query layer: Spark, Trino/Presto, Snowflake, Databricks SQL, Flink, and DuckDB — any engine that implements the table format's read/write spec. Multiple engines can query (and in many setups write to) the same tables concurrently, which is the headline capability worth calling out explicitly in your diagram.

Capabilities the table format layer unlocks

These are the specific technical guarantees that separate a lakehouse from a plain data lake, and they belong as annotations on the table format layer of your diagram rather than buried in prose:

  • ACID transactions on object storage: Atomic commits mean a reader never sees a partially-written table, even while a writer is mid-append — solved via atomic pointer swaps to the latest metadata file, not database locking.
  • Schema evolution without rewriting data: Columns can be added, renamed, or reordered by updating metadata only — existing Parquet files are never rewritten just to reflect a schema change.
  • Time travel / snapshot queries: Every commit creates a new immutable snapshot, so a query can target "the table as of snapshot N" or a specific timestamp — used for reproducible ML training sets and auditing.
  • Partition evolution: A table's partitioning strategy can change over time (for example, from daily to hourly partitions) without rewriting historical data or breaking existing queries.
  • Hidden partitioning (Iceberg): Iceberg derives partition values from column expressions automatically, so queries filter on the logical column (e.g. event_time) without users needing to know or specify the physical partition scheme.

Medallion architecture: Bronze, Silver, Gold

Almost every lakehouse organizes its tables into three internal layers, commonly called the medallion architecture. This pattern belongs inside the storage/table-format portion of your diagram as its own sub-flow, showing data quality improving as it moves left to right.

  • Bronze: Raw data landed as-is from source systems — untransformed, append-only, kept for full history and reprocessing.
  • Silver: Cleaned, deduplicated, and conformed data — type casting, null handling, and business key resolution applied, joined across sources where relevant.
  • Gold: Business-ready, aggregated tables optimized for a specific consumption pattern — dashboards, ML feature sets, or reporting marts. This is the layer BI tools and analysts query directly.

Architecture patterns with prompt templates

Multi-engine lakehouse on S3 with Iceberg

"Multi-engine lakehouse architecture on AWS S3 using Apache Iceberg. Streaming ingestion: application events flow through Kafka, processed by a Flink job that writes directly to Iceberg tables in the Bronze layer (append-only, partitioned by event date). Storage: all Parquet data files live in a single S3 bucket, organized into bronze/, silver/, and gold/ prefixes. Table format: Apache Iceberg manages schema and snapshots for every table across all three medallion layers. Catalog: an Iceberg REST Catalog (Tabular-hosted) is the single source of truth every engine queries to resolve table locations — no engine talks to S3 metadata directly. Medallion flow: Bronze (raw Kafka events via Flink) → Silver (deduplicated, schema-conformed via a nightly Spark job) → Gold (aggregated business metrics tables). Compute: both Snowflake and Databricks read and write the same Gold-layer Iceberg tables concurrently through the shared REST catalog — Snowflake for BI dashboards, Databricks for ML feature engineering. Show the REST catalog as a central hub with arrows to both Snowflake and Databricks, label each medallion layer with its write engine, and annotate the catalog as the shared metadata source of truth."

Migrating a legacy data warehouse to a Delta Lake lakehouse

"Migration architecture from a legacy on-prem data warehouse to a Delta Lake lakehouse on Azure. Source: an on-prem SQL Server OLTP database. CDC: Debezium captures row-level changes from SQL Server's transaction log and publishes them to Kafka topics, one topic per source table. Ingestion: a Databricks Structured Streaming job consumes the Debezium Kafka topics and writes Bronze-layer Delta tables (raw, append-only change events) to Azure Data Lake Storage (ADLS Gen2). Transformation: scheduled Databricks jobs apply merge (upsert) logic to produce Silver-layer Delta tables reflecting current state, then aggregate into Gold-layer business marts. Governance: Unity Catalog manages all table metadata, enforces row- and column-level access policies, and is the single catalog every consumer queries through. Consumers: Databricks SQL serves ad-hoc analyst queries directly against Gold tables; Power BI connects to Databricks SQL Warehouses via the Databricks Power BI connector for exec dashboards. Show the legacy SQL Server as the migration source, the CDC/Debezium path into Bronze, the Bronze → Silver → Gold merge flow, Unity Catalog as a governance layer spanning all three medallion layers, and Power BI plus Databricks SQL as the two downstream consumers."

What to show in your lakehouse architecture diagram

  • Layer separation: Draw storage, table format, catalog, and compute as visually distinct layers, not a single "data lake" blob — the separability of these layers is the entire point of the pattern.
  • Medallion flow: Show Bronze → Silver → Gold as a left-to-right (or top-to-bottom) progression, labeling the transformation applied at each hop (dedup, type casting, aggregation).
  • Which engines read/write which layers: Label every arrow between a compute engine and a table with whether it is a read, a write, or both — this is what makes the multi-engine claim of a lakehouse concrete rather than aspirational.
  • Catalog as shared source of truth: Every engine should have an arrow through the catalog, not a direct arrow to storage — this makes explicit that table location resolution is centralized even though compute is distributed.
  • Governance and access-control annotations: Note where row-level and column-level security is enforced (Unity Catalog permissions, AWS Lake Formation policies) and which roles or groups can access which medallion layer — Gold is typically the widest-access layer, Bronze the most restricted.

Frequently asked questions

Do I have to pick one table format for my whole lakehouse?

Not necessarily. Many organizations standardize on one format for governance simplicity, but interoperability features — Delta UniForm, the spread of the Iceberg REST Catalog spec, and Hudi's growing REST catalog support — increasingly let teams write in one format and read in another without a conversion job. If your organization is multi-vendor by necessity (say, Snowflake for BI and Databricks for ML), diagram the catalog layer carefully, since that is where format compatibility is actually negotiated.

Is a lakehouse the same thing as a data lake with a warehouse bolted on?

No — that describes a two-tier architecture where data is copied from a lake into a separate warehouse for querying, which is exactly what the lakehouse pattern replaces. In a true lakehouse there is one copy of the data, in open file format, in object storage; the table format and catalog layers make that single copy directly queryable with warehouse-grade guarantees, eliminating the ETL hop and the duplicate-storage cost.

Where does the catalog fit versus the table format?

The table format (Iceberg, Delta, Hudi) defines how an individual table's metadata is structured on disk — its snapshots, manifests, and schema history. The catalog is a separate, higher-level service that maps a table name (like sales.orders) to the current location of that table's metadata, and often enforces access control on top. An engine always talks to the catalog first to find a table, then reads the table format metadata directly from storage — draw these as two separate boxes with the compute layer's arrow passing through the catalog.

Related guides: data mesh architecture diagram, dbt architecture diagram, modern data stack architecture, and streaming data architecture diagram.

Ready to try it yourself?

Start Creating - Free