Short answer: CTAS and RTAS are atomic in Spark with Iceberg when you use SparkCatalog, because that catalog implementation stages the metadata change and commits it in a single transition. SparkSessionCatalog does not provide the same atomic commit behavior. RTAS, when atomic, replaces the current table pointer but preserves snapshot history, which means you can recover an earlier state if snapshots are still retained. These statements reflect the documented behavior of Iceberg for Spark and the Iceberg table format, as of the Iceberg documentation and spec checked on November 9, 2026.
Iceberg separates data files and table metadata. Writes produce new data files and then create or rewrite metadata files that point to those data files. Atomic CTAS and RTAS depend on how the metadata file updates are staged and committed. When a catalog coordinates metadata updates by writing temporary metadata and then atomically moving the table pointer, the operation appears atomic to clients. When a catalog does not coordinate that transition, partial-visible states can appear, such as a new table directory with data files but no committed metadata pointer, or a table pointer that points to a partially written metadata file.
Iceberg documents the mechanics for Spark in its Spark DDL and writes documentation, which explains how table metadata and commits are performed for different catalog types. See the Iceberg Spark DDL documentation for the DDL semantics and the Iceberg Spark writes guide for write behavior. The Iceberg spec describes snapshot and manifest behavior that underpins RTAS snapshot history (sources cited below).
Sketch: writer produces data files, writer produces a new metadata draft that references those data files, catalog atomically swaps the table metadata pointer to the new metadata draft. If the swap succeeds, clients see the new table. If the swap fails and the catalog never pointed at the new metadata draft, the table remains unchanged, and temporary metadata and data files can be garbage collected later.
Spark exposes multiple catalog integrations for Iceberg. The two most commonly discussed are SparkCatalog and SparkSessionCatalog. They differ in how they manage metadata locations, staging, and commits.
SparkCatalog is an Iceberg catalog implementation that integrates with external catalog stores (for example a REST catalog, Hive Metastore, or Glue). When configured to do the metadata pointer swap as an atomic operation using the catalog store API, SparkCatalog can stage metadata and then commit the table metadata change as one atomic transition. Because the catalog controls the table pointer, concurrent readers see either the old or the new table state, not an intermediate state.
SparkSessionCatalog is the Spark default catalog that treats table metadata more like a Hive style table on the file system. In many deployments it writes metadata files to a filesystem location and relies on eventual visibility of those files. That does not, by itself, guarantee the same atomic swap semantics as a catalog that provides a compare-and-set or single-pointer update to the table location. In practice, CTAS or RTAS using SparkSessionCatalog may expose transient states where data files or metadata drafts exist but the catalog pointer is inconsistent.
Use this diagram to visualize where the pointer-swap happens for SparkCatalog and where it is absent or weaker for SparkSessionCatalog.
Replace-table-as-select rewrites the table metadata so that the current table snapshot points to a new set of manifests and data files, but Iceberg retains earlier snapshots unless they are explicitly expired or removed by retention settings. That means you get a recovery path back to previous snapshots, subject to retention and garbage collection rules. RTAS changes the table pointer while preserving the snapshot chain so you can roll back by restoring an older snapshot id or by creating a new table state using a historical snapshot.
Important caution, stated clearly: RTAS snapshot history is only useful while snapshots remain retained. If retention policies or explicit expire-snapshots operations have removed the historical snapshots, you cannot recover the prior state from Iceberg metadata. Always verify retention settings before relying on RTAS history for rollback.
CTAS does not infer every property from the source table. You must declare partitioning deliberately if you need it. Many users expect CTAS to copy partitioning and table properties exactly. In Spark with Iceberg, CTAS creates a new table using the SELECT output schema and default properties unless you explicitly specify partitioning, format, or other table-level options. That means CTAS can produce a table with a different partition layout than the source, which affects file sizes, pruning, and read performance.
Practical rule: always specify partitioning and table properties in your CTAS DDL if you care about physical layout. If you need an exact structural copy you can create the table from the source schema and then run an INSERT INTO SELECT, rather than CTAS, to ensure the table metadata is explicit and deliberate.
This section shows an example Spark configuration for SparkCatalog. Verify your Spark and Iceberg versions against the official Iceberg Spark DDL documentation before applying, because configuration keys and catalog capabilities can vary by version (facts checked on November 9, 2026).
Example steps below assume a REST catalog or a metastore-based catalog that supports atomic pointer updates. Replace the example values with your catalog endpoint and credentials. The goal is to direct SparkCatalog to use an external catalog implementation so metadata pointer swaps are coordinated by the catalog.
<!--
spark.conf.set("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.my_catalog.catalog-impl", "org.apache.iceberg.rest.RESTCatalog")
spark.conf.set("spark.sql.catalog.my_catalog.uri", "https://catalog.example.com/api/v1")
-->Notes on the snippet above: the catalog implementation key points SparkCatalog at an implementation that can provide atomic metadata updates, such as a REST-based catalog. If you use a Hive Metastore or Glue, substitute the documented implementation class and configuration keys. See the Iceberg Spark DDL documentation for the exact keys and for capabilities that vary by catalog type.
For readers using Dremio documentation, the Dremio blog post explaining the Apache Iceberg REST catalog is useful to understand how a catalog like RESTCatalog works and why catalog-managed pointer swaps are atomic when the catalog supports them. That post gives deployment context and examples that complement the generic Iceberg docs linked in Sources.
These examples assume Spark in your shell or notebook is already configured with the SparkCatalog named my_catalog from the previous section. They use SQL DDL compatible with Iceberg as implemented for Spark. Verify the DDL options in the Iceberg Spark DDL docs for your version.
<!--
-- CTAS: create a new table from a query result, specifying partitioning and format
spark.sql("CREATE TABLE my_catalog.db.my_ctas_table USING iceberg PARTITIONED BY (country, dt) AS SELECT * FROM source_db.source_table WHERE dt > '2026-01-01'")
-- RTAS: replace the table contents atomically, keeping snapshot history
spark.sql("REPLACE TABLE my_catalog.db.my_ctas_table USING iceberg AS SELECT * FROM another_source WHERE dt > '2026-01-01'")
-->Behavior to verify after each statement: with SparkCatalog, the CREATE TABLE ... AS SELECT should appear in the catalog only after the metadata pointer swap completes. The REPLACE TABLE ... AS SELECT should update the table pointer to a new snapshot while keeping previous snapshots. Check the Iceberg metadata or the table history (time travel) to confirm older snapshots are available.
For inspecting metadata, the Iceberg metadata tables expose internals. If you want a Dremio-specific explanation of querying metadata tables and what to look for in manifests and snapshots, see the Dremio blog post on Iceberg metadata tables and querying the internals. It shows how to read the metadata tables to confirm manifests, snapshots, and file lists after your CTAS or RTAS run.
To test atomicity, inject a failure between data file creation and metadata commit and verify whether the table is visible to readers. The following sequence can be used in a test cluster, for example Apache Iceberg Partition Statistics: compute_partition_stats, Pruning, and Query Planning or Parquet Bloom Filters in Apache Iceberg: Configuration, Selectivity, and Row-Group Skipping. Use a controlled environment only; do not run destructive tests on production without approval.
Test sequence outline:
1. Start a CTAS job that writes data files to the table location but pause the job or block the metadata commit. You can do this by instrumenting the catalog or by using a network rule that blocks the catalog endpoint briefly. If you are using a REST catalog you can return a failure for the commit call. Run this on a test cluster such as Apache Iceberg Partition Statistics: compute_partition_stats, Pruning, and Query Planning so you can safely control the catalog endpoint.
2. While the job is paused after data file creation, try listing the table from another session. With an atomic catalog, other sessions should not see the new table until the metadata commit completes. With a non-atomic catalog, you might see partial files or a directory that looks like a new table but lacks a valid committed metadata pointer.
3. Allow the metadata commit to fail or to be canceled. Observe whether the table pointer changed. If the catalog is atomic, the pointer either remains at the old state or the new table is absent. If the catalog is non-atomic, you may need to run a cleanup pass to remove unregistered data files and metadata drafts. Confirm using the Iceberg metadata tables or the catalog API that no new snapshot is visible.
4. Now allow a successful run. The metadata commit should swap the pointer and the table should appear in queries. For RTAS, verify that the new snapshot id is present and that the older snapshot id is still listed in the table history unless you have explicitly expired snapshots.
Before you use CTAS or RTAS in production, run this checklist in a staging environment.
Expect the following failure modes and design for them.
After you deploy CTAS and RTAS in production, collect these metrics and audit signals regularly.
When deciding whether to use CTAS or RTAS with a given catalog, run this evaluation sequence.
Avoid using CTAS as a shortcut when you need guaranteed copy of all source table properties unless you explicitly set those properties. Avoid RTAS if your retention policy purges snapshots aggressively and you require rollback protection. If your catalog does not support atomic pointer updates, prefer CREATE TABLE then INSERT INTO, which lets you control the commit step more explicitly and lets you validate the table before you switch production traffic.
No. CTAS atomicity depends on which catalog you use. SparkCatalog implementations that coordinate metadata commits can provide atomic CTAS, but SparkSessionCatalog does not provide the same guarantee. Verify your catalog type and implementation against the Iceberg Spark DDL documentation (facts checked on November 9, 2026).
Usually yes, RTAS preserves snapshot history so you can revert to an earlier snapshot id, provided those snapshots have not been expired or garbage collected. Confirm snapshot retention settings before relying on this for recovery.
Run a CTAS while blocking the catalog commit call, then query the table from a separate session. If no partial state is visible until commit completes, the catalog is acting atomically. Use controlled failure injection on a test cluster, such as Apache Iceberg Partition Statistics: compute_partition_stats, Pruning, and Query Planning or Parquet Bloom Filters in Apache Iceberg: Configuration, Selectivity, and Row-Group Skipping.
If the metadata commit never completes, data files and temporary metadata may be orphaned. You must run a cleanup process to remove unreferenced files, using the catalog API or metadata table listings to find files not referenced by any snapshot manifests.
No. CTAS produces a table based on the SELECT output schema and the table creation options you supply. If you need the same partitioning, declare it explicitly in the CTAS DDL or create the table separately and run an INSERT INTO.
Start with the Iceberg Spark DDL guide and the Iceberg Spark writes documentation to understand write and commit mechanics. For the table format and snapshot model, read the Iceberg spec. Links to all three are provided in the Sources section. Also read the Dremio posts I referenced for REST catalog behavior and for how to inspect metadata tables.
Continue with these related implementation guides.
For a deeper treatment, download Apache Iceberg: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.
To put the catalog and table-format ideas into practice, explore Dremio's Apache Iceberg platform.
Additional Dremio resources referenced: an explanation of the Apache Iceberg REST catalog to understand RESTCatalog behavior and a guide to querying Iceberg metadata tables for validation and debugging. These are linked earlier in context where they help explain catalog behavior and metadata inspection: Apache Iceberg REST catalog, what it is and how to use it, Apache Iceberg metadata tables, querying the internals, and a practical guide to Iceberg from Dremio at Dremio Apache Iceberg guide.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
Iceberg keeps per-file bounds automatically but table-level NDV statistics only when you compute them. Why that decides join order, and how to keep them fresh.
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
