Alex Merced

These are not competing choices. One describes what your platform is made of, the other describes how you organise tables inside it. The costly decision is the copy count.

PyIceberg reads and writes Iceberg from Python with no JVM. Here is how to point it at Apache Polaris, and the metadata work that makes it worth keeping installed.

Flink writes Iceberg tables through a catalog. Here is how to wire it to Apache Polaris, and why the checkpoint interval is the most consequential setting in the whole pipeline.

Orphan files are invisible to snapshot expiration and can quietly quadruple your storage. Here is how cleanup finds them, and how to run it without deleting a live table.

Snapshot expiration is what makes a delete in Apache Iceberg reduce your storage bill. Here is what it removes, what it refuses to remove, and how to run it without losing rollback.

A manifest file is an Avro file listing data files that belong to an Apache Iceberg table, along with per-column statistics for each one. Manifests are why Iceberg can plan a query without listing directories: the engine reads partition summaries in the manifest list to skip whole manifests, then reads column bounds inside the survivors […]

Apache Polaris implements the Iceberg REST Catalog specification, a set of HTTP endpoints that any Iceberg client can speak. The specification covers namespace and table management, an atomic commit protocol built on assertions rather than locks, server-side scan planning, and three different ways for a client to obtain storage access. Because the contract is the […]

Choosing an Apache Iceberg catalog comes down to three questions, and the feature matrices most vendors publish do not help with any of them. Do you need more than one engine to write to the same tables? Do you need governance you do not control to be predictable? And do you need to keep the […]

Apache Polaris controls access through role-based access control with four levels: a privilege is granted to a catalog role, the catalog role is granted to a principal role, and the principal role is assigned to a principal. Privileges never attach to a person directly. The two privileges that matter most, TABLE_READ_DATA and TABLE_WRITE_DATA, are the […]

Apache Polaris is a catalog service for Apache Iceberg. It answers the question “which files make up this table right now, and are you allowed to touch them?” Its architecture has four moving parts: an entity model that nests catalogs, namespaces and tables inside a realm, a persistence layer that stores those entities and their […]

Five Apache projects sit underneath most open lakehouses running in production today. Four of them were there a year ago. The fifth, Apache Ossie, entered the Apache Incubator in June 2026 and has not cut its first release yet. That is the fastest way to describe what changed this year. The stack did not get […]

This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]

This is Part 14 of a 15-part Apache Iceberg Masterclass. Part 13 covered streaming approaches. This article is a practical walkthrough of working with Iceberg on Dremio Cloud, covering table creation, data ingestion, optimization, semantic layer construction, and AI-powered analytics. Table of Contents Getting Started Running this walkthrough: Dremio Cloud is no longer self-serve. To use it, contact the […]

This is Part 13 of a 15-part Apache Iceberg Masterclass. Part 12 covered Python and MPP engines. This article covers the three primary approaches to streaming data into Iceberg tables and the operational trade-offs each creates. Iceberg was designed for batch analytics, but most production data arrives continuously. Streaming ingestion bridges this gap by committing data to Iceberg […]

This is Part 12 of a 15-part Apache Iceberg Masterclass. Part 11 covered metadata tables. This article covers the two main ways to access Iceberg data: directly from Python libraries and through MPP (massively parallel processing) query engines. Table of Contents The Python Ecosystem for Iceberg PyIceberg: Native Python Access PyIceberg is the official Python library for Apache […]

This is Part 11 of a 15-part Apache Iceberg Masterclass. Part 10 covered maintenance operations. This article covers the metadata tables that let you inspect Iceberg table internals using standard SQL. Iceberg exposes its internal metadata as queryable virtual tables. You can use them to check table health, debug performance issues, audit changes, and build monitoring dashboards. No […]

This is Part 10 of a 15-part Apache Iceberg Masterclass. Part 9 covered how tables degrade. This article covers the four maintenance operations that keep Iceberg tables healthy and the three approaches to running them. Table of Contents The Four Maintenance Operations 1. Compaction (File Rewriting) Compaction reads small files, merges them into optimally-sized files (128-512 MB), and […]

This is Part 9 of a 15-part Apache Iceberg Masterclass. Part 8 covered embedded catalogs. This article explains the five ways Iceberg table storage degrades and how to detect each problem before it impacts query performance. An Iceberg table that works well on day one will not work well on day 365 without maintenance. Every append, update, and […]

This is Part 8 of a 15-part Apache Iceberg Masterclass. Part 7 covered the traditional catalog landscape. This article examines a newer approach: embedding the catalog directly inside the storage layer. Traditional Iceberg architectures have three components: the query engine, a standalone catalog, and object storage. Embedded catalogs collapse the catalog into the storage layer itself, reducing the […]

This is Part 7 of a 15-part Apache Iceberg Masterclass. Part 6 covered the write process and explained how the catalog enables atomic commits. This article covers what catalogs are, why they matter, and how to choose between the many options available in 2026. A lakehouse catalog is the component that answers one question: “Where is the current […]

Enterprise data rarely sits in one place. It spreads across a warehouse, a few databases, an object store, and a handful of SaaS tools, each with its own access rules. Enterprise agentic analytics is the practice of letting AI agents work across all of that, running the multi-step analysis a human analyst used to do […]

Most enterprise data teams are buried. A business user files a ticket for a number, waits three days, gets a dashboard that almost answers the question, then files another ticket to adjust it. Enterprise agentic analytics breaks that loop by putting AI agents to work directly on governed data, so the person who has the […]

This is Part 6 of a 15-part Apache Iceberg Masterclass. Part 5 covered hidden partitioning. This article walks through the exact steps an engine takes when writing data to an Iceberg table, when the write becomes visible, and how concurrent writers are handled. Understanding the write process is critical because it explains why Iceberg can provide ACID guarantees […]

The Agentic Lakehouse solves a problem that most data teams haven’t fully articulated yet: the architecture you built for human analysts is the wrong architecture for AI agents. A traditional lakehouse is optimized for predictable SQL from BI tools, tuned by DBAs who know the query patterns, and governed through access controls that assume a human is […]

The BIRD benchmark, the leading academic evaluation for text-to-SQL systems, shows that even the best large language models achieve only 60-70% accuracy on complex SQL queries against realistic database schemas. On simple, single-table lookups, accuracy approaches 90%. On multi-join, multi-condition queries that require real business context, it falls off a cliff. That gap is the starting […]

These two terms show up in almost every data platform conversation, and they are often used interchangeably. Both deal with metadata. Both aim to make data more understandable to humans and machines. But they serve fundamentally different purposes, sit in completely different places in your architecture, and solving one problem does not solve the other. […]

This is Part 5 of a 15-part Apache Iceberg Masterclass. Part 4 covered partition evolution. This article covers hidden partitioning, the feature that ensures users never need to know how their data is physically organized. The most expensive mistake in data lake querying is the accidental full table scan: a query that reads every file because the user […]

AI agents can execute hundreds of queries per minute, with no human reviewing each result before the agent acts on it. That is the governance gap that most data architecture teams have not yet closed. Traditional access controls were designed for a world where a person ran a report, read the output, and made a […]

An AI agent that confidently returns the wrong revenue number is more dangerous than one that returns no number at all. Wrong answers that look plausible get acted on. They end up in board decks, budget decisions, and quarterly reports before anyone notices the refunds were never excluded. This is the real problem with AI […]

Every AI agent that touches enterprise data without a standard protocol needs its own custom integration code: bespoke authentication, hand-rolled schema discovery, one-off SQL generation tuned to a specific database. Multiply that by a dozen agents and half a dozen data sources, and you have an integration backlog that grows faster than your team can […]