Dremio is a lakehouse platform that enables companies to run enterprise analytics workloads directly on data lake storage. As part of their data lifecycle, companies can ensure optimal query performance with Dremio’s optimization capabilities for Apache Iceberg tables.
Today, we’re excited to announce Dremio’s support for automated table cleanup, which helps companies easily minimize storage utilization and adhere to data retention policies by removing snapshot, metadata, and data files that are no longer needed.
Snapshots are a fundamental concept in Iceberg. Snapshots help query engines quickly understand which data files comprise a table at a point in time, and are also useful for time travel and rollback scenarios. However, each write to an Iceberg table creates a new snapshot, or version, of that table. These snapshots accumulate over time, and therefore need regular cleanup to minimize table metadata.
Companies can manually expire snapshots and delete unused data files for individual tables using Dremio’s VACUUM TABLE SQL command. However, this can be an arduous task for companies with hundreds or thousands of tables, who need to run the VACUUM TABLE SQL command manually, or write custom schedulers to run this programmatically.
Dremio Software: Companies can now use the VACUUM CATALOG SQL command to expire snapshots and orphaned metadata and data files for all Iceberg tables in a specified catalog. This eliminates the time and effort required to run individual VACUUM TABLE commands against individual tables. VACUUM CATALOG is supported for Nessie Catalogs as of Dremio Software version 24.3.
Dremio Cloud: Automatic table cleanup is now enabled by default for any Dremio Cloud organization using Dremio Arctic as their Iceberg catalog. Arctic automatically performs table cleanup once a day, and deletes orphaned Iceberg metadata files, as well as Iceberg snapshots (i.e., versions) that are older than the customer-defined retention period. When snapshots are deleted, Arctic deletes both the metadata and all Parquet data files that are not referenced by any other snapshot that has not been deleted.
If you have any questions or feedback, please post to the Dremio Community page or contact your Dremio account team and we’ll be happy to assist.
For a deeper treatment of the topics in this post, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.
To see these ideas applied in practice, explore the Dremio documentation.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]
