The Dremio Blog

DuckDB and iceberg-rust can both use an Iceberg REST catalog such as Polaris without a JVM. DuckDB offers SQL-first local analytics through its Iceberg extension. iceberg-rust provides native Rust APIs for applications and services. They share the REST contract and object storage, but their feature coverage and operational roles differ enough that compatibility must be tested separately.

An Iceberg Kafka Connect sink writes Kafka records into Iceberg data files and commits them through a catalog. With Polaris, the connector uses the Iceberg REST interface for table metadata and OAuth-based authorization. A production setup must align topic routing, schema evolution, commit intervals, dead-letter handling, storage credentials, and maintenance because connector health alone does not prove that table snapshots are advancing.

Moving from AWS Glue or Amazon S3 Tables to Polaris is a catalog migration, a permissions migration, and sometimes a storage migration. Existing Iceberg data can often remain in place when Polaris can register the table metadata and access its location. S3 Tables require a separate decision because table buckets are managed resources with their own endpoints, permissions, and service behavior. Do not treat the two source systems as one cutover path.

Iceberg v3 support must be evaluated by feature, operation, and engine version. A product may open a v3 table yet reject deletion vectors, geometry, variant values, encryption, or writes. This matrix records documented support as of September 22, 2026 and separates table-level read, table-level write, and individual v3 capabilities so teams can choose the lowest common denominator deliberately.

Vendor lock-in is the cost and risk of changing a provider, not the presence of a contract or a proprietary feature. The Exit Cost Test measures that cost across data format, catalog metadata, identity and policy, SQL and APIs, operations, egress, and organizational retraining. A platform is easier to exit when another engine can read the same bytes, discover the same tables, reproduce controls, and take over writes in a tested amount of time.

A production Polaris deployment needs a durable relational persistence layer, repeatable schema initialization and migration, externalized secrets, TLS, health probes, metrics, backups, and a tested upgrade path. The official Helm chart supplies Kubernetes resources, but production readiness comes from the values, database operations, identity configuration, and failure testing around the chart.

Polaris event listeners turn catalog and security activity into records that an external logging system can retain and analyze. A production design captures who acted, what resource changed, the result, request identifiers, and timing, while excluding credentials and limiting sensitive metadata. Audit value comes from durable export, normalized identity, retention, and tested alerts, not from enabling a listener alone.

Apache Iceberg 1.12 and Iceberg Rust 0.11 are close to the finish line, and neither one is a routine patch. Apache Iceberg 1.12.0 for Java carries 210 merged pull requests on its milestone. Iceberg Rust 0.11.0 packs roughly two months of work, including end-to-end table encryption and Variant support. Both went through release candidate trouble in mid-September 2026, which tells you something about how much changed.

A data lakehouse is an architecture: object storage, an open table format such as Apache Iceberg, a catalog, and multiple engines reading the same tables. Medallion architecture is a layering convention: bronze for raw data, silver for cleaned data, gold for business-facing aggregates. They are not alternatives. You run medallion inside a lakehouse, and the question worth arguing about is not which to pick but how many of those layers need to be physical copies.

PyIceberg is a pure Python implementation of the Apache Iceberg table format. It talks to Apache Polaris over the Iceberg REST Catalog API, authenticates with an OAuth client credential, and reads and writes tables through Arrow with no JVM and no cluster. It is the right tool for metadata inspection, scripted maintenance, and filtered reads that fit in memory. It is the wrong tool for a distributed join.

Apache Flink writes to Apache Iceberg tables through a catalog, and Apache Polaris is an Iceberg REST Catalog implementation, so Flink connects to it with 'catalog-type'='rest' and an OAuth client credential. Polaris authenticates the job, enforces what it may write, and hands back short-lived storage credentials. Every successful Flink checkpoint becomes one Iceberg snapshot, which is the single most important thing to understand before putting this in production.

An orphan file is a file sitting in an Apache Iceberg table's storage location that no snapshot references, past or present. It is usually the debris of a write that failed partway through. Snapshot expiration cannot remove it, because expiration works from the metadata outwards and the metadata has never heard of it. Finding orphans requires listing storage and comparing that listing against everything the metadata knows about.

Snapshot expiration is the Apache Iceberg maintenance operation that removes old table versions from metadata and deletes the data files only those versions referenced. It is the reason a delete statement eventually reduces your storage bill. It does not delete anything a retained snapshot still needs, and it costs you the ability to time travel to whatever it removed.

A manifest file is an Avro file listing data files that belong to an Apache Iceberg table, along with per-column statistics for each one. Manifests are why Iceberg can plan a query without listing directories: the engine reads partition summaries in the manifest list to skip whole manifests, then reads column bounds inside the survivors to skip individual files. Neither stage touches your data.

Apache Polaris implements the Iceberg REST Catalog specification, a set of HTTP endpoints that any Iceberg client can speak. The specification covers namespace and table management, an atomic commit protocol built on assertions rather than locks, server-side scan planning, and three different ways for a client to obtain storage access. Because the contract is the specification rather than a vendor SDK, an engine written against one implementation works against all of them.