This article has been revised and updated from its original version published in 2022 to reflect the latest Apache Iceberg developments, including V3 features, the REST Catalog specification, and the rise of Apache Polaris.
Apache Iceberg is an open table format for large-scale analytic datasets. It sits between your query engine (Spark, Flink, Dremio, Trino, DuckDB) and your data files (Parquet, ORC, Avro) on object storage. Iceberg adds the transactional guarantees, performance optimizations, and management capabilities that raw file formats lack.
If you're hearing terms like "data lakehouse," "open table format," or "REST Catalog" and want to understand what they mean in practice, this guide covers everything from architecture to hands-on tutorials.
Traditional data lakes store files in directories on object storage. To run a query, the engine lists those directories to find the right files, a process that becomes painfully slow as tables grow to thousands or millions of files.
Hive tables improved on this by tracking partitions in a metastore, but they still had major limitations:
Apache Iceberg was created at Netflix to solve all of these problems. It was donated to the Apache Software Foundation in 2018 and has since become the most widely adopted open table format, supported by more engines than any competitor.
Explore this interactive demo and see how Dremio's Intelligent Lakehouse enables Agentic AI
Iceberg replaces directory listings with a structured metadata tree. Understanding this tree is the key to understanding how Iceberg delivers performance and reliability.
Catalog → Metadata File → Manifest List → Manifest Files → Data Files
To understand this tree in depth, read Hands-On with the Iceberg Table Structure and The Life of a Read Query for Apache Iceberg Tables.
This layered metadata structure gives Iceberg its signature performance advantage:
| Pruning Level | What It Uses | What It Skips |
|---|---|---|
| Manifest list | Partition summaries | Entire manifest files whose partitions don't match the query |
| Manifest file | Column min/max stats | Individual data files that can't contain matching rows |
| Parquet row group | Row group statistics | Row groups within a file that don't match |
A query that might scan 10,000 files in a traditional data lake might scan only 50 files after Iceberg's three-level pruning. For a detailed look on how this works, see How Apache Iceberg Is Built for Performance.
Every write to an Iceberg table creates a new snapshot atomically. Readers always see a consistent view of the table, either the old snapshot or the new one, never a partial write. This is possible because commits are atomic metadata pointer swaps via optimistic concurrency control.
Iceberg tracks columns by unique ID, not by position or name. This means you can:
int → long)All without rewriting data files. Existing data files remain valid. The engine simply maps the old schema to the new one at read time.
In traditional Hive-style partitioning, users must include partition columns in their queries. Iceberg eliminates this with hidden partitioning: you define partition transforms (like month(order_date)), and Iceberg automatically maps query predicates to the right partitions. Users write WHERE order_date = '2024-03-15' and Iceberg handles the rest.
Need to change your partition scheme as data grows? Iceberg supports partition evolution, letting you change from month(ts) to day(ts) without rewriting existing data. Old data stays in monthly partitions, new data goes into daily partitions, and queries spanning both work correctly.
Every snapshot is immutable. You can query any previous version of your table by snapshot ID or timestamp:
SELECT * FROM orders FOR SYSTEM_TIME AS OF '2024-01-15';
This is essential for auditing, debugging, and reproducible analytics.
When a bad data ingestion corrupts your table, rollback reverts to a previous snapshot instantly. It's a metadata-only operation: no data files are deleted or rewritten.
Iceberg supports row-level changes through two strategies: copy-on-write (rewrites entire files) and merge-on-read (writes small delete files). V3 introduces deletion vectors for even faster merge-on-read writes.
| Version | Key Additions |
|---|---|
| V1 | Core spec: metadata tree, schema evolution, partitioning, snapshots |
| V2 | Row-level deletes (positional + equality delete files), partition stats |
| V3 | Deletion vectors, variant type, default values, row lineage |
V3's deletion vectors are a significant performance improvement. They replace separate delete files with bitmap-based markers stored in Puffin files, giving you fast writes and fast reads. For the full V2-to-V3 comparison, see the table format architecture detailed look and the official Iceberg specification.
The catalog is the single source of truth for table metadata location. Iceberg supports multiple catalog implementations:
| Catalog | Key Feature |
|---|---|
| REST Catalog (standard) | Engine-agnostic HTTP API, credential vending |
| Apache Polaris (Incubating) | Open-source REST catalog with RBAC |
| Project Nessie | Git-like branching, multi-table transactions |
| AWS Glue | Managed catalog for AWS environments |
| Hive Metastore | Legacy compatibility |
For the full history, see The Evolution of Apache Iceberg Catalogs.
Any engine that implements the Iceberg library or REST Catalog client can read and write Iceberg tables:
Python developers have three ways to work with Iceberg: PyIceberg (pure Python, no JVM), PySpark (distributed processing), and Arrow Flight (SQL via Dremio).
This 12-part video series walks through every core concept. Watch them in order for the best experience:
Iceberg tables need periodic maintenance to stay performant. Three operations matter:
Small files accumulate from streaming ingestion or frequent small writes. Compaction rewrites them into larger, optimally sized files (256-512MB target).
Every commit creates a new snapshot. Over time, hundreds of snapshots accumulate. Expiring old snapshots reclaims metadata space.
Failed writes or expired snapshots can leave unreferenced data files on storage. Orphan cleanup removes them.
Dremio automates all three operations for tables managed by its catalog, so you don't need to schedule them manually.
There are multiple paths to get data into Iceberg:
| Method | Best For | Related Guide |
|---|---|---|
| Dremio CTAS | Converting files already on storage | CSV to Iceberg, JSON to Iceberg |
| Spark SQL | Large-scale batch ingestion | Introduction to Iceberg Using Spark |
| Flink | Streaming ingestion | Flink + Iceberg + Nessie |
| Airbyte/Fivetran | CDC from databases | Lakehouse with Airbyte, Lakehouse with Fivetran |
| AWS Kinesis + Glue | AWS streaming | Kinesis + Glue → Iceberg |
| Format migration | Moving from Hive or Delta | Hive to Iceberg, Delta to Iceberg |
Apache Iceberg is one of three major open table formats, alongside Delta Lake and Apache Hudi. The architecture comparison and feature comparison cover the differences in detail. Key differentiators for the current spec version:
| Factor | Iceberg | Delta Lake | Hudi |
|---|---|---|---|
| Engine support | Broadest (15+ engines) | Spark-centric + UniForm | Spark/Flink |
| Catalog standard | REST Catalog spec | Unity Catalog | Hudi metadata |
| Governance | ASF (multi-vendor) | Linux Foundation | ASF |
| Partitioning | Hidden + evolution | Liquid Clustering | Traditional |
| Community | Largest contributor base | Databricks-led | Uber-originated |
Use Iceberg when:
Iceberg may not be ideal when:
The fastest way to get hands-on with Iceberg:
The fastest path to hands-on experience is through Dremio Cloud. Sign up for a free trial, connect to your object storage, and create your first Iceberg table with a simple CREATE TABLE statement. Dremio handles all the catalog management, compaction, and query optimization automatically. For local development, use Docker with Spark and a REST catalog like Nessie or Apache Polaris.
No. Iceberg is engine-agnostic by design. Any engine that implements the Iceberg library or connects via the REST Catalog API can read and write the same tables. You can write data with Spark, query it with Dremio, and analyze it with DuckDB, all against the same table.
Iceberg tracks every column by a unique integer ID rather than by name or position. When you add, drop, rename, or reorder columns, existing data files remain untouched. The engine maps old schemas to new schemas at read time using those IDs, so queries against historical snapshots still return correct results.
Upgrade to V3 when your tables have frequent row-level updates or deletes. V3's deletion vectors provide significantly faster write and read performance for those operations compared to V2's positional delete files. If your tables are append-only, V2 is sufficient.
After watching the series of videos above, you should have a pretty good understanding of Apache Iceberg and its concepts.
Below is a list of additional resources to continue learning more about Apache Iceberg, including hands-on exercises, articles from companies detailing their usage of Apache Iceberg and more.
Below are several resources for understanding what Apache Iceberg is and how it fundamentally works at a high-level conceptual level.
Below are resources to learn more about the many features of Apache Iceberg.
The resources below guide you through guided exercises and tutorials to try Apache Iceberg in action with different tools.
Videos showing hands use of Apache Iceberg Tables
With the resources below you can read on how Apache Iceberg compares to other table formats.
Below are articles from companies that have documented their deployment of Apache Iceberg into production. You can read about their experiences and lessons learned.
Once you have Apache Iceberg tables in place you’ll want to optimize and maintain them, below are articles that walk through different features for engineering tables for best performance.
How do we get data into our Iceberg tables, the following are articles on the ingestion of data into Iceberg tables from different sources.
Object storage has become the standard for storing data in a data lakehouse and the resources below highlight Apache Iceberg in the context of cloud object storage.
Below are articles on Apache Iceberg’s Java and Python API.
Streaming data can require lots of considerations that don’t exist in batch processing. Below are resources that deal with using Apache Iceberg in streaming data.
Take your Apache Iceberg tables to the next level with Project Nessie/Dremio Arctic catalog, which allows you to create catalog-level branches for isolating ETL, catalog rollback, multi-table transactions, and more. Here are some talks and blogs on the subject.
Recordings of Apache Iceberg Office Hours, held as part of the Gnarly Data Waves podcast.
Here is a list of other great Apache Iceberg articles you can learn from.
Episodes of the Gnarly Data Waves Podcast dedicated to Apache Iceberg, subscribe on Youtube or Spotify.
Here is a list of Subsurface conference talks on Apache Iceberg.
Even more talks from Subsurface Live 2023!
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
We're always looking for ways to better handle and save money on our data. That's why the "data lakehouse" is becoming so popular. It offers a mix of the flexibility of data lakes and the ease of use and performance of data warehouses. The goal? Make data handling easier and cheaper. So, how do we […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
Dremio Arctic bring new features to Dremio Cloud, including Apache Iceberg table optimization and Data as Code.
