Incremental processing with Apache Iceberg reads a defined snapshot range and records the last successfully applied snapshot as a durable checkpoint. For append-only flows a snapshot-range append scan is sufficient. When records are updated or deleted you must use changelog-capable views or engine-specific change processing. A dependable pipeline explicitly handles exclusive start and inclusive end snapshot semantics, snapshot expiration, branch ancestry, checkpoint durability, and idempotent replay logic.
Iceberg organizes table history as a sequence of snapshots. Each snapshot references a set of data files and points to a manifest list that lists file additions and deletions. Incremental processing means selecting a contiguous range of snapshots to read and applying the deltas from that range to an external system. The common pattern is: record the last-applied snapshot id as the checkpoint, on the next run read from checkpoint+1 through current snapshot id, apply changes, then update the checkpoint after a durable commit.
Two important technical cautions to keep top of mind, because they cause obvious failures if you forget them. First, with many engines, notably Apache Spark, incremental reads are append-only. That means Spark-only incremental code will not see deletes or updates as record-level change events. Second, Iceberg snapshot range semantics are start-exclusive and end-inclusive. If your checkpoint records snapshot N, the next run must start from N+1. Missing that off-by-one causes duplicates or gaps.
Think of checkpoints as a simple pointer into the snapshot timeline. At time T0 the checkpoint is snapshot 42. A new writer creates snapshots 43, 44, 45. The incrementer reads snapshots 43..45, applies the deltas, and atomically writes checkpoint 45. If an error occurs mid-apply, the checkpoint remains 42 and the pipeline retries from 43. The checkpoint must be durable and written only after a successful sink commit. Common failure modes include the checkpoint being written before the sink commit, or concurrent writers advancing the table while the pipeline is mid-apply.
Append scans read only file additions in a snapshot range. They are cheap because the engine can ignore deleted-file markers. Changelog scans produce row-level change events, such as insert, update, delete. The latter requires the table metadata to expose precise change information and the query engine to support it. Trino, Iceberg native readers, and some engines expose change data through specialized reads or metadata tables; Spark incremental workflows are append-only unless you explicitly use Iceberg change-table features or compute diff from metadata tables.
Practically, use append scans for append-only sources like event streams that are only ever appended. Use changelog scans when the upstream writes include UPDATE or DELETE operations, for example when you run CDC or mutable dimension tables. The difference is not academic, it affects whether your sink needs to be able to upsert or delete, and whether you must preserve order within a partition during application.
Exactly-once across an Iceberg incremental pipeline requires two things: atomic checkpointing and an idempotent or transactional sink. Atomic checkpointing means the pipeline writes the checkpoint only after the sink confirms the changes are durable. An idempotent sink means that reapplying the same snapshot range yields the same external state, or the sink can deduplicate by sequence id or snapshot id.
Common approaches to idempotency: include snapshot id metadata on each row or batch, and have the sink ignore batches that are already applied. Another pattern is to use a transactional sink (for example a database transaction or a Kafka producer with exactly-once semantics) that commits the entire batch and the checkpoint together under the same durable transaction. If your sink cannot do that, you must design idempotent application logic in the sink or a deduplication layer at read time.
Checkpoint durability is critical. A common implementation stores checkpoint state in an external durable store, for example a relational table, a durable key-value store, or the same transactional system used by the sink. The implementation must guarantee an atomic transition: the snapshot range is applied to the sink and the checkpoint value is recorded such that either both become visible or neither does.
<!--
Pseudo-Java-like pseudocode for a durable checkpoint transaction. Replace with your platform transaction primitives.
-->
try {
beginTransaction(); // sink + checkpoint store support a two-phase or single transaction
List<Snapshot> snapshots = listSnapshots(fromExclusive, toInclusive);
applySnapshotsToSink(snapshots); // idempotent apply or transactional write
writeCheckpoint(toInclusive); // durable commit marker
commitTransaction();
} catch (Exception e) {
rollbackTransaction();
throw e;
}
Do not invent a custom local file as the only durable checkpoint unless you also store it in durable distributed storage. If a node dies, local state disappears and your pipeline will either reprocess from an older snapshot or fail to resume.
This worked example shows the concrete configuration values you will need for an Iceberg snapshot-range read. The exact API calls differ by engine and version, so verify them against your engine docs. The following points are checked as of October 27, 2026: Iceberg documents snapshot-based scanning and start/end semantics on the project documentation. The examples below are representative and you should map them to your engine API calls.
Required parameters for a snapshot-range read:
Example mapping to engines. Verify API names against the engine documentation for your installed versions. For Spark with Iceberg, the project documents snapshot queries and examples on the Iceberg website. Trino documents connector-level snapshot options in its Iceberg connector documentation.
What to verify before deploying: confirm your engine version supports snapshot range reads and whether it provides changelog semantics. Spark incremental reads remain append-only in documented behavior unless you use explicit change-table or metadata table approaches. Refer to the Iceberg Spark queries documentation for current behavior.
Below is a step-by-step sequence you can implement with a transactional database sink that supports transactions across both data and a checkpoint table. If you lack a single transaction covering both, you must implement a two-phase commit pattern or rely on idempotent sink operations.
If you must write the checkpoint in a separate system from the sink, ensure you design retry semantics so that reprocessing a snapshot range before the checkpoint is idempotent. For example, include snapshot id metadata on all sink rows and have the sink deduplicate by snapshot id and primary key.
Gaps occur when a checkpoint points to snapshot N but the table has advanced to M and you cannot read some intermediate snapshots, most commonly due to snapshot expiration or retention policies. Detecting and recovering from gaps is essential to avoid data loss or inconsistent state.
Detect gap conditions using the following checks:
Replay procedure when a gap is detected:
Iceberg snapshot expiration can invalidate a stale checkpoint. The Iceberg project documents snapshot lifecycle and default retention behavior. Check your table retention policies and any cleanup job that may compact or expire old snapshots. If you rely on incremental processing, extend snapshot retention or export snapshots to cold storage to enable recovery.
Operationally, design your pipeline to detect missing snapshot artifacts early. If the checkpoint is expired you either run a full table copy or use a backup-based restore. A failure mode I have seen in production is a nightly compaction job that reduces snapshot retention to a few days. An incremental consumer that runs weekly suddenly cannot find manifests for older snapshots and fails during the listing step. The right fix was either to increase retention for tables with incremental consumers or to add an archival pipeline that copies manifests to long-term storage accessible to the consumer.
Not every engine exposes the same snapshot or changelog APIs. Two concrete references you must check for your environment: the Iceberg project documents Spark query behavior and snapshot operations on the Iceberg documentation site, and Trino documents its Iceberg connector snapshot and table interaction behavior. Engines may differ on whether they expose row-level delete/update events, whether they support time travel by snapshot id, and whether they can produce a changelog stream.
As of October 27, 2026 these references are authoritative for project behavior. Consult the Iceberg releases page and release blog posts for specifics about changes in snapshot handling across versions. If you are using Spark, confirm whether your Spark + Iceberg combination provides changelog semantics or only append scans. I have repeatedly seen teams assume Spark incremental reads include deletes; that assumption causes silent divergence when the source has updates.
Useful links right in this discussion: the Iceberg Spark queries documentation helps you map snapshot read examples to Spark SQL or DataFrame APIs. The Iceberg releases and 1.11.0 release blog post explain recent snapshot and metadata-table features that may be relevant. The Trino Iceberg connector documentation is useful if you use Trino for your reads. Keep these references handy when implementing or upgrading your stack.
After deploying incremental processing you must measure both pipeline health and data correctness. Key metrics and checks:
On top of numeric metrics, add periodic data quality checks: row counts by partition and business key reconciliation between source and sink. If your sink supports materialized views or primary-key validation, schedule an hourly parity check for critical partitions.
Here are the failure modes I see most often, with concrete mitigations.
Use this checklist when you introduce incremental snapshot-based ingestion into production.
Testing must be practical and repeatable. I recommend these experiments in a staging cluster such as the clustering test environment or the multi-engine DML test environment to validate behavior under controlled conditions. Use the clustering test environment for functional tests and the multi-engine DML test environment for scale tests, so you exercise different failure surfaces.
Tests to run:
Automate these tests as part of CI/CD. Include tests that assert the checkpoint semantics, including off-by-one behavior for start exclusive and end inclusive ranges.
When designing a pipeline you will face three practical decisions: append-only versus changelog, where to store the checkpoint, and how to ensure idempotency. Make these choices explicit.
For teams running Dremio as part of their stack, the Dremio guide to Apache Iceberg and the Dremio post on querying Iceberg metadata tables are useful references. The guide explains common configuration choices and shows examples for integrating Iceberg with execution engines. The post on metadata tables explains how to query internal metadata for manifest and snapshot information, which helps for gap detection and offline reconstruction.
Short version, measure correctness, freshness, and stability. These are the concrete metrics I track in production.
Use the monitoring recommendations in the Dremio post on monitoring Iceberg table health to collect manifest-level metrics and snapshot timelines. That post walks through metrics to collect and how to expose them through metadata tables so you can automate gap detection and retention alarms.
Before flipping a production consumer to incremental mode, run this quick checklist as a scripted verification in your staging cluster.
<!--
Pseudo-check script. Replace CLI and API calls with your platform equivalents.
-->
# 1. Check current snapshot
current_snapshot = get_table_current_snapshot(table)
# 2. Read checkpoint
checkpoint = read_checkpoint(checkpoint_store, consumer_id)
# 3. Verify checkpoint not older than retention
earliest_snapshot = get_earliest_available_snapshot(table)
assert checkpoint >= earliest_snapshot, "checkpoint expired"
# 4. Perform a dry-run list
snapshots = list_snapshots(checkpoint, current_snapshot)
validate_snapshots(snapshots)
# 5. Run a single-batch apply and rollback test
apply_snapshots_dry_run(snapshots[0])
If any step fails, do not enable incremental mode. Fix the condition and rerun. I have seen teams switch to incremental inbound consumption only to discover missing manifests at scale; that mistake costs hours of triage and a complete sink resync.
Implement incremental processing by treating snapshots as the unit of work, checkpointing snapshot ids durably after sink commits, and choosing append versus changelog modes based on your source semantics. Make snapshot expiration and checkpoint recovery first-class parts of your design. Test the edge cases, automate gap detection, and use Dremio metadata and monitoring guidance to keep a reliable operational picture.
This section gives a concrete, runnable sequence you can follow to implement the snapshot-range read, durable checkpoint transaction, and gap detection and replay procedure already discussed. The steps assume a Spark job that uses Iceberg snapshot-range reads, a separate durable checkpoint store (for example a transactional metastore or an atomic key value store), and an idempotent sink. The recipe focuses on what to set, what to validate, and what to do when a snapshot expires. Verify behavior against the Iceberg Spark queries documentation and your engine version, because behavior and supported options vary between Iceberg releases and connectors.
High level steps, each expanded below:
All code examples below are pseudocode. Do not assume these exact APIs exist in your environment. Check the Iceberg Spark queries documentation and your engine's connector docs for the exact call signatures and supported options. The examples assume Spark 3.x with the Iceberg Spark runtime and a key value store that supports conditional writes for the checkpoint commit.
Before the first run, record a checkpoint entry that points to the snapshot id you intend to start after. The convention here is the checkpoint stores the last processed snapshot id, so the next run will read snapshots greater than that id. Confirm the checkpointed id exists in the table history or is lower than the earliest retained snapshot. If the checkpoint points to an expired snapshot, you must choose whether to fail, fast forward to the earliest available snapshot, or perform a full refresh. Record the decision in your runbook.
Validation checklist:
Remember, for Iceberg snapshot-range reads: the start snapshot id is exclusive and the end snapshot id is inclusive. That pattern prevents double processing when you persist the last-processed snapshot id as the checkpoint. Use the checkpointed id as start_exclusive and select an end_inclusive based on time, number of commits, or a target snapshot id taken from the table metadata.
Example decision rules to pick the end snapshot id:
Concrete verification before read:
Run the engine read with the computed start and end. For Spark, your call should use the engine-provided snapshot-range read option. Remember Spark incremental reads are append only in Iceberg as of some versions. That means append scans will not directly surface deletes and updates unless the engine supports changelog views or you use a snapshot-range read that includes file level changes and applies delete files. Confirm your engine version and connector implementation first.
Write results to a sink that can deduplicate or is idempotent. Ideal sinks accept an idempotency key per record or support transactional upserts. If you only have append sinks, ensure the sink deduplicates using a stable record key and snapshot id metadata.
After the sink write completes, update the checkpoint atomically. Use a conditional write that moves the checkpoint from the prior value to the new end_inclusive only if the stored value still equals the prior checkpoint. This avoids lost updates when multiple workers compete. If your checkpoint store supports conditional compare-and-swap or single-row database transactions, use them.
Example logic:
<?php
// Pseudocode: conditional commit
old = read_checkpoint(table_id)
if old != expected_old:
raise ConcurrencyError
write_checkpoint_if_equals(table_id, expected_old, new_end_inclusive)
?>If the conditional commit fails, you must detect whether the sink write still corresponds to the committed checkpoint by reading the actual stored checkpoint. If another process moved the checkpoint forward to a value that includes your end_inclusive, you can consider your work already covered and skip further action. If the checkpoint moved to an earlier snapshot or a different branch, you must run the gap detection procedure below.
Gap detection compares the checkpoint progression across time or across partitions. A simple approach uses a sequence of checkpoints with wall clock timestamps. If you expect the checkpoint to advance monotonically, detect any backward moves or holes where the checkpoint was not updated for an expected period. For per-partition processing, verify each partition checkpoint independently.
When gaps are detected, you will replay the missing snapshot range. Use a snapshot-range read that covers the missing inclusive range. If the missing range includes deletes or updates, prefer a changelog-capable read if your engine supports it, or perform a snapshot replay that re-applies file level changes. The replay must be idempotent. One reliable pattern is to include snapshot id and source run id in every written record and let the sink deduplicate by (record key, snapshot id) or keep the latest snapshot id per key.
Edge case: if a checkpoint points to an expired snapshot, you cannot build the missing diffs from metadata. In that case, consider a full table scan or an optimized manifest list scan that reconstructs current state. The choice depends on the data volume and SLA.
Testing failure modes is necessary to gain confidence. Below is a small test plan you can run in a staging cluster. Each test includes the fault injection, the expected behavior, and how to verify. Run these tests against the same engine and Iceberg version you plan to deploy. Some behaviors are version dependent. Verify the observed results against the Iceberg Spark queries documentation and your connector docs.
Test 1: checkpoint commit races
Test 2: mid-read job crash
Test 3: snapshot expiration during gap replay
Test 4: concurrent external compaction or rewrite
Beyond the monitoring already described, add the following actionable metrics that tie directly to snapshot-range processing, checkpoints, and replay reliability. These metrics focus on operability, not just system health.
Suggested alert thresholds are conservative. Adjust them to your workload. Always attach the checkpoint history and snapshot ids to alerts to speed diagnosis. Label metrics with table id, pipeline id, and worker region.
When a checkpoint points to an expired snapshot, you cannot reconstruct diffs using that snapshot id. You have three pragmatic recovery policies. Each has operational trade offs. Pick the one that matches your SLAs and data volume, and codify it in automation and runbooks.
Document the chosen policy for each pipeline, and automate detection so that the system either executes the policy or creates an incident that includes a recommended action and costs for a rebuild.
Use a canary rollout when you introduce snapshot-range processing and durable checkpoints. The plan below gives a small script you can run for each canary batch. It assumes you control a small subset of tenants or partitions that are representative of production traffic.
Verification script pseudocode to run after each canary batch. Replace storage and connector calls with your engine APIs before use:
<?php
// Pseudocode: post-deploy verification for a partition
last_checkpoint = read_checkpoint(pipeline, partition)
snapshots = list_table_snapshots(table)
if last_checkpoint > snapshots.latest:
raise Alert('checkpoint beyond latest snapshot')
if last_checkpoint < snapshots.earliest:
raise Alert('checkpoint points to expired snapshot')
// Validate sink state matches table state at last_checkpoint
sink_count = count_sink_records(partition)
expected_count = count_table_state_at_snapshot(table, last_checkpoint)
if abs(sink_count - expected_count) / expected_count > 0.01:
raise Alert('sink mismatch')
?>Label all verification outputs with the Iceberg version you ran these checks against. Some behaviors, such as the specifics of snapshot-range support or changelog scans, can change between releases. Check the project release notes for changes that affect your pipeline.
This section refines the earlier monitoring section into an operational measurement plan you should run for the first 72 hours, and then continuously at lower frequency. Collect these metrics with labels: table id, pipeline id, partition id, run id, and Iceberg version. Tag each metric with the snapshot range processed (start_exclusive, end_inclusive).
Collecting these metrics during the initial 72 hour ramp will reveal steady state and corner cases. Use the data to tighten checkpoint frequency, snapshot retention, and compaction schedules.
No. Spark incremental reads are append-only in documented behavior for common Spark+Iceberg integrations. To see deletes or updates you must use change-aware features that expose row-level changes, or compute diffs using metadata tables. Verify behavior against the Iceberg Spark queries documentation for your versions.
Start snapshot id is exclusive and end snapshot id is inclusive. If your checkpoint records snapshot N, your next range should start at N+1 and end at the current snapshot id. Off-by-one errors lead to duplicate application or gaps.
If the checkpoint references a snapshot that has been expired and the required manifests or files are no longer available, the incremental consumer cannot reconstruct the missing deltas. You must either run a full resync from the current snapshot or restore archived snapshots. Detect this condition by comparing checkpoint id to the earliest available snapshot id and alert immediately.
Store checkpoints in a durable, highly available store. Prefer a transactional database or the sink itself when it supports transactions. If that is not possible, use a distributed key-value store or a dedicated checkpoint table. The important property is that checkpoint writes are durable and atomic relative to sink commits.
Make replays idempotent by attaching snapshot id metadata to every applied record or batch and having the sink ignore duplicate snapshot ids. Alternatively, use a transactional sink that can rollback and commit batches atomically with the checkpoint write.
The Dremio article on monitoring Iceberg table health explains which metrics and metadata tables to expose for manifest and snapshot monitoring. The Dremio post on querying Iceberg metadata tables shows practical queries you can run against metadata tables to locate manifests, list snapshots, and detect expiration. The Dremio Apache Iceberg guide provides broader integration details and configuration examples for using Iceberg with query engines.
These guides cover adjacent implementation details that are outside this article's main scope.
For a deeper treatment, download Apache Iceberg: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.
To put the catalog and table-format ideas into practice, explore Dremio's Apache Iceberg platform.
Additional Dremio resources cited inline: Dremio post on querying Iceberg metadata tables, Dremio guide to monitoring Iceberg table health, and Dremio Apache Iceberg guide. Use the clustering test environment for staging functional tests and the multi-engine DML test environment for scale tests in the rollout examples.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
Iceberg keeps per-file bounds automatically but table-level NDV statistics only when you compute them. Why that decides join order, and how to keep them fresh.
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.