The Dremio Blog

Parquet and ORC are columnar and built for reading. Avro is row-oriented and built for writing. How to choose, and why Iceberg quietly uses all three.

Parquet is how columnar data sits on disk and Arrow is how it sits in memory. Every difference between them follows from that one split.

Iceberg is a table format and Parquet is a file format. They are different layers of one stack, not alternatives, and an Iceberg table contains Parquet files.

Apache Iceberg 1.12.0 expands v3 engine support, REST catalog operations, streaming reliability, and v4 groundwork. Learn how to upgrade safely.

Apache Polaris 1.8.0 adds semantic-model privileges, safer commit handling, pagination controls, CLI improvements, and important database upgrade steps.

Apache Iceberg v3 defines geometry and geography as first-class primitive types. Values are encoded as WKB in Avro and Parquet, with optional CRS metadata and spatial bounds that engines can use for pruning. The specification is stable, but engine support is version-specific, so teams should test every reader and writer before upgrading a shared table.

The Apache Iceberg View Spec makes view metadata portable by storing versioned schemas, catalog and namespace context, dependencies, properties, and one or more SQL representations. It does not make every SQL expression universally executable. Portability depends on a catalog and engines that implement the spec and agree on a supported dialect representation.

Iceberg table encryption separates file encryption from key management. Writers create data keys, encrypt files, wrap those keys with a key encryption key in a KMS, and store key metadata with the files. Iceberg can encrypt data files, delete files, manifests, and manifest lists. Table metadata JSON is intentionally not encrypted by the Iceberg encryption specification.

Server-side scan planning lets an Iceberg REST catalog plan table scans close to metadata instead of sending every manifest to the client. A client discovers the capability, submits a scan request, receives completed tasks or an asynchronous plan identifier, then polls for batches and can cancel work it no longer needs. The feature reduces metadata transfer, but it moves planning load and observability into the catalog service.

The Iceberg REST protocol defines an optional transaction commit endpoint that can submit updates for several tables as one catalog operation. That endpoint can provide catalog-level atomicity when the server implements it, but a usable multi-table transaction also requires client support, conflict handling, and clear failure semantics. A REST catalog label alone does not guarantee that an engine will issue coordinated commits.

MERGE INTO is the clearest way to express keyed changes against an Iceberg table, but reliable upserts start before the MERGE. Deduplicate the source to one action per key, define late-arrival rules, and make the batch replayable. For SCD Type 2, close the current row and insert a new version in a controlled transaction, then compact the delete and data files the workload creates.

Iceberg storage cost is not only the number of terabytes in data files. It includes PUT, GET, LIST, and DELETE requests, retained snapshots, delete files, manifests, metadata JSON, compaction rewrites, and recovery windows. The safest cost plan starts with table-aware expiration and orphan cleanup, then applies object-store lifecycle rules only to paths and objects whose ownership is understood.

Healthy Iceberg monitoring combines event reports with table-state queries. ScanReport shows planning and filtering behavior. CommitReport records write outcomes and snapshot changes. Metadata tables reveal file sizes, delete-file pressure, snapshot growth, and manifest shape. Together they support SLOs for freshness, commit reliability, planning latency, and maintenance debt.

A Polaris realm is a top-level administrative boundary that lets one Polaris service host separate sets of principals, roles, catalogs, and configuration. Realms reduce accidental cross-tenant visibility at the catalog layer, but they do not replace network isolation, separate encryption keys, storage policies, database controls, or resource quotas. Production tenancy needs all of those boundaries to agree.

Apache Polaris can validate OAuth 2.0 access tokens from an external OpenID Connect provider instead of issuing every identity itself. The reliable setup aligns issuer, audience, JWKS, claim mapping, token lifetime, and Polaris role grants. Keycloak and Okta differ in administration, but Polaris sees the same signed token contract.