Apache Iceberg keeps two different kinds of statistics. Per-file bounds in the manifests, which exist automatically and decide which files a query reads. And table-level statistics in Puffin files, which do not exist until you compute them and decide how a query planner shapes a join. Almost every team has the first and almost nobody has the second.
The symptom is familiar. A three-table join is slower than it should be. The scan looks fine, file pruning is clearly working, and yet the plan picks a join order that makes no sense to anyone who knows the data. The optimizer is not being stupid. It is guessing, because nobody gave it the numbers it needed.
Every data file that Iceberg tracks has an entry in a manifest file recording its partition values, record count, file size, null counts and the minimum and maximum value for each column. These are written at commit time, automatically, with no action from anyone.
Their job is elimination. Given a predicate, an engine uses them to rule out files before opening any of them. They answer “can this file possibly contain matching rows”. The read path walks through how that unfolds.
The second kind describes the table as a whole rather than any one file. The most important is the number of distinct values in a column, usually written NDV, stored in a Puffin file and registered in the table metadata as a statistics file.
Its job is estimation. Given a join or an aggregation, a cost-based optimizer uses NDV to guess how many rows will survive a filter and how large an intermediate result will be. Those guesses determine join order and join algorithm.
Bounds tell an engine what to skip. NDV tells it what plan to build. Neither substitutes for the other.
Consider joining a billion-row fact table to two dimensions, filtered on both.
An optimizer has to decide which join to run first, and the right answer is whichever produces the smaller intermediate result. To know that, it needs to estimate how many rows survive each filter, which means knowing roughly how many distinct values the filtered column holds. A column with 4 distinct values and a column with 4 million behave completely differently under equality.
Without statistics the optimizer falls back on heuristics, often a fixed selectivity guess. Sometimes that is close enough. On a star schema where one dimension filters to ten rows and another to ten million, it is not, and the plan builds the large intermediate first.
This is why the symptom is so specific. Scans are fine, pruning works, and the join order is wrong. Nothing about file-level statistics addresses it.
Iceberg ships a Spark procedure for this.
-- all top-level primitive columns, current snapshot
CALL catalog.system.compute_table_stats('db.events');
-- a specific snapshot
CALL catalog.system.compute_table_stats(
table => 'db.events', snapshot_id => 3821553639163471111);
-- only the columns that matter
CALL catalog.system.compute_table_stats(
table => 'db.events', columns => array('customer_id', 'product_id'));By default it computes statistics for every top-level primitive column using the current snapshot. On a wide table that is more work than you need. The columns worth computing are the ones that appear in join conditions and group-by clauses, and restricting to those makes the job dramatically cheaper.
The result is a Puffin file registered against the snapshot it was computed from.
Puffin is Iceberg's format for statistics and index blobs. The relevant blob type here is apache-datasketches-theta-v1, which holds a Theta sketch and exposes an ndv property carrying the distinct value estimate derived from it.
A sketch rather than an exact count, and that is the point. Counting distinct values exactly across a billion rows means holding every value seen. A Theta sketch holds a bounded sample and estimates from it, with a small and known error. An optimizer choosing between join orders does not need an exact number, it needs the right order of magnitude. The Puffin format also carries deletion vectors in v3 tables, which is a separate use of the same container.
A second procedure computes statistics per partition rather than per table.
CALL catalog.system.compute_partition_stats('db.events');
CALL catalog.system.compute_partition_stats(
table => 'db.events', snapshot_id => 3821553639163471111);This one is incremental. It computes from the last snapshot that has a partition statistics file up to the given snapshot, and only does a full computation when no previous file exists. On a large table that difference is the difference between a job you can run nightly and one you cannot.
Partition statistics help an optimizer reason about skew. A table partitioned by region where one region holds 80 percent of the rows plans very differently from one where the regions are even, and partition-level counts are what make that visible.
This is the operational catch, and it is the reason most teams who compute statistics once stop benefiting from them within a month.
A statistics file is registered against the snapshot it was computed from. The table keeps committing. Nothing invalidates the file, nothing warns anyone, and the numbers drift from reality at whatever rate the data changes.
Stale statistics can be worse than none. A missing statistic makes an optimizer fall back to a heuristic it knows is a guess. A stale one gives it a number it trusts, and confident wrong estimates produce worse plans than acknowledged uncertainty.
Recompute on a cadence that matches how fast the distribution changes, not how fast the table grows. Appending more of the same data barely moves NDV. Onboarding a new customer segment, a new region or a new product line moves it immediately. A table that doubles overnight with similar data may need no update, while one that gains a single new tenant may need one straight away.
A workable default for most warehouses is weekly, run after compaction, and on demand after any bulk load that changes the shape of the data rather than its volume.
The metadata tables expose all of this, so there is no guessing about whether statistics exist. Querying the internals covers the full set.
-- which snapshots have statistics files, and for which columns
SELECT * FROM catalog.db.events.metadata_log_entries;
-- current snapshot, to compare against what statistics were built from
SELECT snapshot_id, committed_at, operation
FROM catalog.db.events.snapshots
ORDER BY committed_at DESC LIMIT 5;
-- per-file bounds, which exist whether or not you computed anything
SELECT file_path, record_count, null_value_counts, lower_bounds, upper_bounds
FROM catalog.db.events.files LIMIT 10;The comparison that matters is between the snapshot a statistics file was computed from and the current snapshot. A large gap, measured in commits rather than in time, is the signal to recompute.
One honest caveat before anyone schedules a nightly job.
Writing statistics is standard. Iceberg defines the Puffin format and the procedures to produce them. Using them is an engine decision, and engines differ in whether they read Iceberg table statistics, whether they maintain their own, and which parts of the plan they influence.
Check your engine before concluding the statistics did nothing. The reliable test is empirical: capture a plan for a representative join, compute statistics, capture the plan again, and compare the join order and the estimated row counts. If neither moved, your engine is not using them and the job is wasted effort until it does.
That test takes ten minutes and saves scheduling something that produces no benefit.
It is worth placing all three statistical layers next to each other, because they are constantly confused.
The first two reduce how much you read. The third decides what you do with it. A table can be perfectly pruned and still produce a bad plan, which is exactly the case where people conclude Iceberg is slow.
Dremio maintains its own statistics and reflections for query acceleration, and reads Iceberg tables natively. The general point holds regardless of engine: the statistics an optimizer uses are only as good as the last time anyone computed them, and a table that has changed shape since then is planning against a picture of the past.
Because Puffin statistics live in the table rather than in an engine, computing them once makes them available to any engine that reads them, the same way compaction benefits every reader rather than the engine that ran it.
Take the slowest join in your warehouse. Capture its plan, note the join order and the estimated row counts, then compute statistics on just the join key columns and capture the plan again.
If the order changes, you have found free performance and a job worth scheduling. If nothing changes, you have learned something equally useful about your engine in ten minutes rather than after a month of nightly runs.
Per-file statistics yes, table-level statistics no. Manifests record column bounds, null counts and record counts at commit time automatically. Distinct value estimates must be computed explicitly with compute_table_stats.
The estimated number of distinct values in a column. A cost-based optimizer uses it to estimate how many rows survive a filter and how large an intermediate result will be, which determines join order and join algorithm. Without it the optimizer guesses.
Iceberg's container format for statistics and index blobs, stored beside the data and registered in table metadata. The relevant blob type for statistics is apache-datasketches-theta-v1, which carries a Theta sketch and an ndv estimate. Puffin also carries deletion vectors in v3 tables.
Counting distinct values exactly across a very large table means tracking every value seen. A Theta sketch holds a bounded sample and estimates from it with known error. An optimizer needs the right order of magnitude, not an exact figure.
On a cadence that matches how fast the data distribution changes, not how fast the table grows. Appending more of the same data barely moves NDV. A new tenant, region or product line moves it immediately. Weekly after compaction suits most warehouses.
Yes, and they can be worse than having none. A missing statistic makes an optimizer fall back on a heuristic it treats as a guess. A stale one gives it a number it trusts, and a confident wrong estimate produces a worse plan than acknowledged uncertainty.
No. Writing them is standard, using them is an engine decision. Test empirically: capture a plan for a representative join, compute statistics, capture it again and compare join order and estimated row counts. If nothing moves, your engine is not reading them.
For a full treatment of the table format, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.
To see these ideas applied in practice, explore the Dremio documentation.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
Apache Iceberg 1.12.0 expands v3 engine support, REST catalog operations, streaming reliability, and v4 groundwork. Learn how to upgrade safely.