The Dremio Blog

A data lakehouse is an architecture: object storage, an open table format such as Apache Iceberg, a catalog, and multiple engines reading the same tables. Medallion architecture is a layering convention: bronze for raw data, silver for cleaned data, gold for business-facing aggregates. They are not alternatives. You run medallion inside a lakehouse, and the question worth arguing about is not which to pick but how many of those layers need to be physical copies.

PyIceberg is a pure Python implementation of the Apache Iceberg table format. It talks to Apache Polaris over the Iceberg REST Catalog API, authenticates with an OAuth client credential, and reads and writes tables through Arrow with no JVM and no cluster. It is the right tool for metadata inspection, scripted maintenance, and filtered reads that fit in memory. It is the wrong tool for a distributed join.

Apache Flink writes to Apache Iceberg tables through a catalog, and Apache Polaris is an Iceberg REST Catalog implementation, so Flink connects to it with 'catalog-type'='rest' and an OAuth client credential. Polaris authenticates the job, enforces what it may write, and hands back short-lived storage credentials. Every successful Flink checkpoint becomes one Iceberg snapshot, which is the single most important thing to understand before putting this in production.

An orphan file is a file sitting in an Apache Iceberg table's storage location that no snapshot references, past or present. It is usually the debris of a write that failed partway through. Snapshot expiration cannot remove it, because expiration works from the metadata outwards and the metadata has never heard of it. Finding orphans requires listing storage and comparing that listing against everything the metadata knows about.

Snapshot expiration is the Apache Iceberg maintenance operation that removes old table versions from metadata and deletes the data files only those versions referenced. It is the reason a delete statement eventually reduces your storage bill. It does not delete anything a retained snapshot still needs, and it costs you the ability to time travel to whatever it removed.

A manifest file is an Avro file listing data files that belong to an Apache Iceberg table, along with per-column statistics for each one. Manifests are why Iceberg can plan a query without listing directories: the engine reads partition summaries in the manifest list to skip whole manifests, then reads column bounds inside the survivors to skip individual files. Neither stage touches your data.

Apache Polaris implements the Iceberg REST Catalog specification, a set of HTTP endpoints that any Iceberg client can speak. The specification covers namespace and table management, an atomic commit protocol built on assertions rather than locks, server-side scan planning, and three different ways for a client to obtain storage access. Because the contract is the specification rather than a vendor SDK, an engine written against one implementation works against all of them.

Choosing an Apache Iceberg catalog comes down to three questions, and the feature matrices most vendors publish do not help with any of them. Do you need more than one engine to write to the same tables? Do you need governance you do not control to be predictable? And do you need to keep the catalogs you already run while you migrate? Answer those and the field narrows from seven options to about two.

Apache Polaris controls access through role-based access control with four levels: a privilege is granted to a catalog role, the catalog role is granted to a principal role, and the principal role is assigned to a principal. Privileges never attach to a person directly. The two privileges that matter most, TABLE_READ_DATA and TABLE_WRITE_DATA, are the only ones that cause Polaris to issue short-lived storage credentials, and they sit outside the metadata grant that covers everything else.

Apache Polaris is a catalog service for Apache Iceberg. It answers the question “which files make up this table right now, and are you allowed to touch them?” Its architecture has four moving parts: an entity model that nests catalogs, namespaces and tables inside a realm, a persistence layer that stores those entities and their grants, a REST service that speaks the Iceberg REST Catalog API, and a credential broker that hands engines short-lived, path-scoped access to object storage.

The state of the open lakehouse in September 2026 is a stack that stopped arguing about whether it won and started dealing with the consequences of winning.

This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]

This article is a practical walkthrough of working with Iceberg on Dremio Cloud, covering table creation, data ingestion, optimization, semantic layer construction, and AI-powered analytics.

Streaming ingestion bridges this gap by committing data to Iceberg tables at regular intervals. The challenge is that frequent commits create the small file problem, and managing that trade-off between data freshness and table health is the central concern of streaming to Iceberg.

This article covers the two main ways to access Iceberg data: directly from Python libraries and through MPP (massively parallel processing) query engines.