Apache Iceberg Concurrency: Commit Conflicts, Isolation and Retries

Apache Iceberg uses optimistic concurrency. Writers do not lock anything, they prepare a commit and then attempt an atomic swap. If another writer got there first, the commit is rejected and retried against the new state. Most production concurrency problems are not bugs, they are default retry settings meeting a workload nobody tuned them for.

Two Spark jobs write to the same table. One fails after twenty minutes with a commit conflict. The table is fine, no data was lost, and the on-call engineer still has to work out whether to change the schedule, the isolation level or the partitioning.

This covers what a conflict actually is, the properties that govern retries, the two isolation levels and what each one checks, and the design choices that stop conflicts happening at all.

The Model: Nobody Holds a Lock

A writer reads the current table state, does its work, and then tries to commit by swapping the metadata pointer from the version it started with to a new one. The swap is a compare-and-swap. How that mechanism works is a post in itself, and the relevant part here is that it either succeeds or it does not.

Nothing is locked while the writer works. Two jobs can spend an hour writing files to the same table in parallel, and both sets of files are real and durable. The contention is compressed into one instant at the end.

This is a deliberate trade. Locking a table for the duration of a write would serialise every job and make a slow write block everything behind it. Optimistic concurrency lets writers overlap and only pays a cost when they genuinely collide.

What a Conflict Actually Is

A conflict is not two writers touching the same table. That is normal and usually fine. A conflict is a writer discovering that the assumption its work was based on is no longer true.

A pure append rarely conflicts. Adding new files does not depend on which files already exist, so when the pointer has moved the engine can usually rebase: take the new current state, add the same new files, commit again. The second attempt succeeds.

A delete, update or merge is different. It read a set of files, decided which rows to change, and produced output based on that decision. If another writer removed or replaced one of those files in the meantime, the output is based on a state that no longer exists, and committing it would lose the other writer's change. That one has to fail.

So the pattern is predictable. Appends coexist happily. Row-level operations conflict with other row-level operations and with maintenance on the same files.

The Retry Properties

Iceberg retries rejected commits automatically, and the defaults are set for a modest amount of contention.

commit.retry.num-retries              4
commit.retry.min-wait-ms              100
commit.retry.max-wait-ms              60000        -- 1 minute
commit.retry.total-timeout-ms         1800000      -- 30 minutes

commit.status-check.num-retries       3
commit.status-check.min-wait-ms       1000
commit.status-check.max-wait-ms       60000
commit.status-check.total-timeout-ms  1800000

Four retries with exponential backoff between 100 ms and a minute, giving up after 30 minutes in total. On a table with a nightly batch load and an hourly stream, that is comfortable. On a table with six concurrent writers committing every thirty seconds, four retries is not many.

Raising commit.retry.num-retries is the obvious first move and it is often the right one. It is also a treatment rather than a cure. If a job needs twelve retries to land, the writers are fighting over the same files and the layout is the real problem.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'commit.retry.num-retries' = '10',
  'commit.retry.min-wait-ms' = '200'
);

The status check, and why it exists

The second group of properties covers a nastier case. A writer sends the commit and the connection drops before it learns the outcome. The commit may have succeeded or it may not, and retrying blindly risks applying the same change twice.

Rather than guess, Iceberg re-reads the table to find out whether its commit landed. commit.status-check.num-retries governs how many times it checks before giving up and reporting an unknown commit state.

An unknown commit state is the one failure mode worth alerting on separately. It does not mean the write failed, it means nobody knows, and the correct response is to look at the table rather than to rerun the job.

Isolation Levels

Iceberg offers two isolation levels for row-level operations, set per table and per operation type.

write.delete.isolation-level   serializable   -- default
write.update.isolation-level   serializable   -- default
write.merge.isolation-level    serializable   -- default

Serializable is the default and the stricter of the two. A commit is rejected if another writer added or deleted files in the partitions the operation touched. It guarantees the result is as though the operations ran one after another.

Snapshot isolation only checks for concurrent deletes in those partitions. Concurrent appends are tolerated. It is weaker, and the specific thing it gives up is worth understanding before choosing it.

Consider a DELETE FROM events WHERE status = 'expired' running while a stream appends new rows, some of them already expired. Under serializable, the delete fails because rows it should have considered arrived while it worked. Under snapshot, it commits, and the newly arrived expired rows survive. No corruption, no error, just a delete that did less than its predicate suggests.

Snapshot isolation is the right choice when that outcome is acceptable, which it often is for retention jobs that run repeatedly. It is the wrong choice when a delete is a compliance action and has to be complete.

ALTER TABLE catalog.db.events SET TBLPROPERTIES (
  'write.delete.isolation-level' = 'snapshot'
);

In the Spark DataFrame API the same control appears as an isolation-level write option, alongside validate-from-snapshot-id for pinning the snapshot that conflict detection measures against. Setting it to null disables the checks entirely, which is only safe for genuinely idempotent writes.

Copy-on-Write Makes Conflicts More Likely

The write mode interacts with concurrency directly, and this catches people.

write.delete.mode   copy-on-write   -- default
write.update.mode   copy-on-write   -- default
write.merge.mode    copy-on-write   -- default

Under copy-on-write, changing one row rewrites the entire data file containing it. That file is now replaced, so any other operation that read it is invalidated. A small update can invalidate a large amount of other writers' work.

Under merge-on-read, the data file is left alone and a small delete file is written beside it. Far less is replaced, so far less conflicts. The cost moves to read time, where delete files must be applied, and to maintenance, where they must be compacted away. The full trade-off is its own topic.

For a table with frequent concurrent row-level writes, switching to merge-on-read is often a more effective conflict fix than raising the retry count.

Designing Conflicts Out

Retry tuning treats symptoms. These address the cause.

Partition disjointness

Conflict detection is partition-scoped. Two writers working on different partitions do not conflict, however busy the table is. If your writers can be arranged so each owns a partition range, by region, tenant, or date, the problem disappears structurally.

This is the single highest-value change available and it is usually a scheduling decision rather than a code change.

Separate ingestion from mutation

Appends and row-level operations rarely fight. Row-level operations fight each other constantly. Batching updates into one job rather than running several concurrently removes most of the contention at a stroke.

Schedule maintenance away from writers

Compaction rewrites files, which invalidates anything that read them. Running it against a table being actively updated produces exactly the conflicts you would expect. Give it a window, or scope it to partitions that are no longer being written. Polaris policies are one way to express that.

Keep write transactions short

The conflict window is the time between reading the table state and committing. A job that reads, spends forty minutes computing, then commits has a forty-minute window. One that computes first and reads late has a much smaller one.

Diagnosing a Conflict in Production

When a job fails with a commit conflict, four questions resolve it quickly.

• What else was writing? Query the .snapshots metadata table and look at the commits around the failure, including their operation type and summary.
• Were they the same partitions? If not, the conflict is not what it appears and the isolation level is probably stricter than it needs to be.
• Was maintenance running? A compaction or expiry overlapping a writer is the most common cause and the easiest to fix.
• Did it exhaust retries or time out? Exhausted retries on a busy table suggests raising the count. A timeout suggests the commit itself is slow, which is a different problem.

SELECT committed_at, snapshot_id, operation, summary
FROM catalog.db.events.snapshots
ORDER BY committed_at DESC
LIMIT 20;

Where Dremio Fits

Dremio reads and writes Iceberg tables natively and runs table maintenance against the tables it manages. Because the commit protocol belongs to the Iceberg specification rather than to any engine, a commit from Dremio contends with a commit from Spark exactly as two Spark commits would, and the same isolation and retry properties apply.

That is the point of the specification. Concurrency is a property of the table, not a feature each engine implements its own way, which is what makes multiple engines writing the same table safe rather than merely possible.

The Setting Worth Checking Today

Open your busiest table and read two properties: commit.retry.num-retries and write.delete.mode. If the first is still 4 and the second is still copy-on-write while several jobs update the table concurrently, you have found the explanation for most of your failed commits.

Neither change requires rewriting data, and both can be reversed.

Frequently Asked Questions

Can two engines write to the same Iceberg table at once?

Yes. The commit protocol is part of the Iceberg specification, so a Spark writer and a Dremio writer contend exactly as two Spark writers would. Appends generally coexist, and row-level operations on the same partitions conflict.

What causes a commit conflict?

A writer committing work that was based on files another writer has since replaced or removed. Pure appends rarely conflict because they do not depend on which files already exist. Deletes, updates and merges do.

How many times does Iceberg retry a commit?

Four by default, with backoff from 100 ms up to a minute and a 30 minute overall timeout. Raise commit.retry.num-retries on tables with several concurrent writers, but treat a need for many retries as a sign the layout is the real problem.

What is the difference between serializable and snapshot isolation?

Serializable, the default, rejects a commit if another writer added or deleted files in the affected partitions. Snapshot only checks for concurrent deletes, so it tolerates concurrent appends. Snapshot produces fewer conflicts and allows a delete to miss rows that arrived while it ran.

Does merge-on-read reduce conflicts?

Yes, usually significantly. Copy-on-write rewrites whole data files, invalidating anything else that read them. Merge-on-read writes small delete files instead, so much less is replaced. The cost moves to read time and to compaction.

What is an unknown commit state?

The connection dropped before the writer learned whether its commit succeeded. Iceberg re-reads the table to check, governed by the commit.status-check.* properties. If it still cannot tell, it reports an unknown state. Inspect the table rather than rerunning the job.

Will a conflict corrupt my table?

No. A rejected commit changes nothing. The data files it wrote remain in storage unreferenced, which is why orphan file cleanup exists, but the table itself moves from one consistent snapshot to the next.

Keep learning

For a full treatment of the table format, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.

To see these ideas applied in practice, explore the Dremio documentation.