Dremio is now part of SAP
Dremio Blog

34 minute read · November 17, 2026

Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino

Testing Apache Iceberg Across Dremio, Spark, Flink, and Trino
Copied to clipboard

An Apache Iceberg support matrix tells you what an engine claims to implement. A compatibility contract proves that your exact engine versions, catalog, storage layer, and table format produce the same committed state and query results.

TL;DR

  • From each engine, perform a controlled append of N distinct rows, with a deterministic primary key and a monotonically increasing event_time column. Use small N for CI and larger N for stress runs.
  • After each append, capture the table snapshot id, snapshot timestamp, and the manifest list location; store these in artifacts for later inspection.
  • From every engine, run a point-in-time read of each snapshot id using the engine's time travel syntax or API, and assert the row counts and a checksum of deterministic columns match the writer's expectations.

Illustrative checks (do not assume universal syntax):

  • assert snapshot count increased by one after an append
  • assert row_count(snapshot) == sum of appended rows up to that snapshot
  • assert deterministic checksum of (id,event_time) matches writer's checksum

Example failure mode: one engine commits with a different partition spec than expected. The snapshot exists, but time-travel queries return different record distributions or miss records due to a partition-spec mismatch. Investigate the manifest files and the partition-spec field in the snapshot manifest to locate the divergence.

Schema, partition, and sort evolution contract

Goal: Validate evolution operations do not violate Iceberg compatibility rules and that reads from prior snapshots continue to function.

  • Define a baseline schema with nested types if you use nested data. Reference Dremio's schema evolution guide for rules that apply to nested structures; also consult the Iceberg spec for naming and id assignment semantics. Note: schema evolution implementations differ by engine and by Iceberg format version.
  • Run schema additions that are always compatible, such as adding optional columns, and verify old snapshots and queries still succeed. Record schema ids from the table's current metadata and ensure they increment as the spec prescribes.
  • Run incompatible changes (for example changing a column from int to string) only in separate negative tests that assert engines reject the change or produce expected errors.
  • For partition evolution, test adding and removing partitions in controlled steps and assert that the partition-spec id in metadata changes in the way the spec documents.
  • For sort evolution, if engines support ordered writes at file level, verify the new sort order is honored for performance tests, but do not rely on identical file layouts across engines.

Failure mode: two engines assign different field ids when creating columns with the same name because the creation sequence and tooling differ. To avoid this, prefer schema operations done through the catalog-backed alter table operations rather than ad-hoc table recreation, and record the field ids in the manifest for each evolution step.

Delete, retry, and commit-conflict tests

Goal: Exercise delete semantics, retry behavior, and conflict resolution by forcing concurrent commits and retried attempts.

  • Delete tests: from each engine, execute row-level deletes or delete-file operations the engine supports. Record the resulting snapshots and verify that time travel to the pre-delete snapshot returns the original rows, while queries against the latest snapshot omit deleted rows.
  • Retry tests: simulate client retries by issuing the same write operation multiple times from the same engine with identical identifiers where the engine supports idempotent retries. Verify that only a single logical write appears or that the behavior matches the engine's documented retry semantics. Record the commit metadata entries that indicate retry handling.
  • Commit-conflict tests: run two concurrent writers (either two different engines or same engine with separate processes) that try to modify overlapping data, and observe whether one commit fails with a concurrent modification error or one writer retries and succeeds. Record the exact error messages and the catalog behavior.

Example failure mode: an engine's connector retries in a way that produces duplicate rows when the client retries at the application layer. To diagnose, compare the row-level identifiers, examine manifest file lists, and check for repeated file additions in successive snapshots. If you see duplicate data, investigate the engine's write-idempotence documentation in the engine-specific Iceberg write docs.

Failure injection and release gate

Introduce deterministic failure scenarios and treat successful completion of the failure set as the release gate for cross-engine write interoperability. Keep the experimental v3 gate separate. For each injected failure, you must capture artifacts: writer stack traces, catalog logs, object-store operation logs, and the table's metadata snapshots.

  • Network partition between engine and catalog during commit, observe atomicity and rollback behavior.
  • Object-store transient failures during file writes, examine whether partial files are left and how subsequent cleanup occurs.
  • Concurrent commit races between two writers targeting the same snapshot.
  • Client crash between manifest upload and snapshot commit.

Failure injection must be reproducible and limited in scope. For example, simulate an S3 5xx for a single file write, not a full-bucket outage, unless you are deliberately testing capacity to recover from global outage. After each failure injection, include reconciliation steps: catalog metadata inspection, aborting in-progress transactions if the engine exposes that capability, and restoring the object store to a consistent state where possible.

Validation plan and operating metrics

Validation has two orthogonal components: metadata invariants and query result equivalence. Record and expose operational metrics for both.

Metadata invariants to record

CROSS-ENGINE CONTRACT ARCHITECTURE1Versioned test manifest2Append and time-travel con…3Schema, partition, and sor…4Delete, retry, and commit-…Each stage produces evidence that can stop the rollout before the next boundary.
Cross-engine contract architecture.
  • Snapshot ids and timestamps after each write. The Iceberg spec defines snapshot id semantics; capture them verbatim. (See the Iceberg specification.)
  • Manifest list locations and the set of manifest files referenced by the snapshot.
  • Partition-spec id and mapping after partition evolution steps.
  • Schema ids and field id assignments after schema evolution steps.
  • Manifest file content summaries such as number of added data files, deleted files count, and per-file record counts. Do not assume byte equality for the manifests between engines; instead compare logical content.

Query result metrics

  • Row count per snapshot for representative queries.
  • Checksums (for example CRC64 or SHA256) of deterministic, stable columns across snapshots.
  • Latency of point-in-time reads for typical query shapes: simple count, filter by partition, join with small dimension table.
  • False negatives/positives for queries that rely on partition pruning or predicate projection.

Operating metrics to track during CI and in pre-release acceptance runs:

  • Test run duration and per-test latency distributions.
  • Number of transient errors and their categories (network, S3, catalog).
  • Commit conflict rate when running concurrent writer tests.
  • Percentage of metadata checks that require manual inspection (indicates specification ambiguity or engine divergence).

Implementation steps: concrete, executable, and environment-aware

Below is a practical checklist you can follow. Where a command depends on engine CLI, I mark it "illustrative". Verify exact syntax in the engine docs linked below. Record outputs to your artifacts location and commit the run manifest.

  • Prepare environment
    • Provision or select a shared object store bucket and ensure all engines have read/write permissions. Record provider, bucket, region in the manifest.
    • Stand up or select a catalog, and ensure all engines use the same catalog for the run. Record catalog type and version in the manifest.
    • Fix engine versions and connector artifacts. Install or stage connectors and record their exact artifact coordinates in the manifest.
  • Create canonical table
    • Choose one engine to create the table and specify the exact create-table options. For example, using Spark (illustrative):
      -- illustrative only:
      -- spark.sql("CREATE TABLE my_catalog.db.t (id INT, data STRING, event_time TIMESTAMP) USING iceberg TBLPROPERTIES ('format-version'='2')")
      Record the engine and creation options in the manifest. If using Dremio to create the table, consult Dremio's Apache Iceberg tables SQL documentation for supported statements and options.
  • Run append and time-travel writes
    • From each engine, append a deterministic set of rows. Capture snapshot id and manifest list after each append. Example illustrative snippet for verification only:
      -- illustrative only:
      -- writer computes checksum = sha256(concat(id,cast(event_time as string)))
      -- writer inserts rows with ids 1..N and records the produced snapshot id after commit
      
    • From each engine, perform time-travel reads to every snapshot id and verify row counts and checksums match the expected values recorded by writers.
  • Schema and partition evolution
    • Apply a compatible schema change such as adding an optional column. Record old and new schema ids and field ids. Verify prior snapshots continue to return their original rows.
    • Apply a partition evolution step, for example adding a day-based partition spec. Capture partition-spec id transitions and validate reads that rely on partition pruning.
  • Delete and conflict tests
    • Run a delete from one engine and verify visibility across engines. Time-travel to pre-delete snapshot to confirm rows are still present there.
    • Simulate concurrent writers that attempt to modify overlapping keys. Observe conflict errors and that only one commit succeeds unless the engine documents automatic conflict resolution.
  • Failure injection
    • Introduce transient object-store errors for a single file write and observe cleanup. Record the manifest files to see partial file entries and how each engine or catalog cleans them up.
    • Isolate a catalog outage during commit and observe whether the writer reports success and how the table state reflects the partial/failed commit.
  • Collect and validate artifacts
    • Store snapshots of the table metadata, manifest files, and engine logs in the artifacts location. Include the manifest and a per-engine run log that includes engine version and connector artifact coordinates.
    • Run a validation script that checks metadata invariants and query results. Report mismatches to a triage dashboard for manual review.

Failure modes to expect and where to look

  • Metadata layout differences: engines may write manifests in different orders or include additional optional metadata fields. Compare logical content, not raw bytes.
  • Partition mismatches: partition spec id or partition-expression differences cause files to be written under unexpected partition folders, producing query differences when engines apply different pruning logic.
  • Field id divergence: adding columns via different tools can assign different field ids; this breaks equality for nested data unless reconciled.
  • Retry semantics causing duplicates: client-side retries combined with non-idempotent engine writes can produce duplicate data. Cross-check file-level additions across snapshots.
  • Catalog-specific atomicity: some catalog implementations may not garantir atomic operations the same way; record catalog logs to inspect commit races.

Validation checklist and automated assertions

WRITE-READ VALIDATION CYCLERisk 1The test suite must record exact engine and connector ver…Test itRisk 2A passed read test does not establish write interoperabil…Bound itRisk 3Metadata equality is not always byte equality because eng…Monitor it
Write-read validation cycle.
  • Manifest exists and is parseable by each engine's Iceberg client.
  • Snapshot id referenced by the latest metadata matches the one announced by the creating engine.
  • Row counts per snapshot match writer expectations across engines.
  • Checksums of stable columns match across engines for the same snapshot id.
  • Partition-spec id and schema id transitions are logged and consistent with spec behavior.
  • For delete and merge tests, the set of deleted file entries and tombstones in manifests match the expected state for the committed snapshot.

Operating metrics to capture continuously

  • Per-run pass/fail status and time to complete.
  • Per-test failure cause categories and counts.
  • Commit conflict frequency and mean time to resolve.
  • Time to reconcile metadata inconsistencies when manual intervention is required.

Version-specific cautions

Iceberg has multiple format versions and experimental features. Exercise caution and record versions as follows.

  • Iceberg format versions: specify the Iceberg format version (for example v1, v2, or v3) you are testing in the manifest. The spec and engine write docs describe what features are associated with each format version. If you enable v3 features, treat them as experimental and gate them explicitly. The Iceberg spec is the primary reference for format semantics.
  • Spark writes: consult the Spark writes guide for behavior specific to your Spark and Iceberg connector versions. The Iceberg project documents Spark-specific write semantics that influence retries and manifest file grouping.
  • Flink writes: Flink's integration has its own semantics, for example how Flink tasks commit parallel writes. Consult the Flink writes documentation when interpreting concurrent write behaviors.
  • Trino connector: Trino's connector has documented limitations and configuration knobs; consult the Trino Iceberg connector documentation for connector-specific behavior, especially for catalog interaction and write modes.
  • Dremio: use Dremio's Apache Iceberg tables SQL documentation for Dremio-specific SQL commands and any differences in how Dremio writes Iceberg metadata. Use the supplied Dremio blog posts for support matrices and DML specifics when planning tests across engines.

FAQ

  • Q1: If a read from Engine A shows the correct rows but a write from Engine B does not, which should I trust?

    A: Trust the Iceberg spec and your manifest. A successful read only proves the reader can parse the visible metadata. It does not prove the writer committed the expected state. Use snapshot ids and the manifest list to compare writer claims against catalog state. If Engine B produced a snapshot id that does not contain the expected file additions, investigate the writer logs and the catalog commit trace.

  • Q2: Can I assume manifest files will be identical across engines?

    A: No. The Iceberg specification allows engines to serialize metadata slightly differently. Always compare logical invariants: file additions, deletions, partition specs, and schema ids. Byte-for-byte equality is not a valid contract unless you control a single writer implementation.

  • Q3: How do I test experimental Iceberg v3 features?

    A: Treat them as a separate gate in your manifest. Only run v3 tests when every engine and catalog in the run documents support for the specific v3 features you need. Record the experimental_v3_gate boolean and the explicit feature flags. Expect failures that would not occur under v2 and require per-engine troubleshooting.

  • Q4: What if two engines assign different field ids when adding the same column name?

    A: This is a common source of incompatibility. Resolve it by performing schema evolution through the same catalog alter operation when possible, or by reconciling field ids in a controlled maintenance step. Record field ids in the manifest and include a reconciliation test that validates id equality across engines.

  • Q5: How can I be confident about deletes and merge semantics?

    A: Do not rely on read-only checks. Run write-based delete and merge tests from each engine that claims support. Verify the resulting committed snapshots for the presence of delete markers, equality of file tombstones, and expected time-travel behavior. Use the Dremio DML support matrix to see claim coverage, then construct your cross-engine tests accordingly.

  • Q6: Does a passed read test mean my table is safe for production multi-engine writes?

    A: No. A passed read test shows read compatibility for the tested snapshot and engine versions, but write interoperability requires passing the full contract: append/overwrite/delete tests, conflict and retry scenarios, and failure injection tests. Only when the run manifest documents that all those tests passed for your exact engine, connector, and catalog versions should you consider multi-engine writes for production.

Worked examples: diagnosing three real failure scenarios

These are short, concrete examples that show how to locate the root cause. Commands are illustrative; consult the source docs for exact CLI syntax.

Scenario 1: Missing rows after Trino append

METADATA INVARIANTS BY FEATUREDeclareversioned contractExecutecontrolled fixtureCompareexpected stateGateship or stopA failed result returns with evidence. It does not become an undocumented exception.
Metadata invariants by feature.

Symptoms: Trino reports successful commit and produces a snapshot id, but Spark and Dremio time-travel reads to that snapshot are missing a subset of rows.

  • Check Trino's writer logs for file write errors that were logged but not surfaced as commit failures.
  • Inspect the snapshot manifest list that Trino published. Compare the list of added files to the actual objects in the object store. If files referenced in manifest are missing, the writer recorded the paths but the upload failed or was never completed.
  • Compare partition-spec id in the snapshot metadata against the table's current partition-spec. If Trino wrote files under an unexpected spec, queries from Spark may not locate them due to different pruning logic.

Resolution: If Trino wrote files with missing uploads, reconcile by re-uploading missing objects or recreating the manifest entries via a repair operation. If partition spec mismatch caused the issue, add a migration step to normalize partition specs through the catalog.

Symptoms: Flink job retried due to checkpoint failure; final table contains duplicate rows for the retried batch.

  • Check Flink write semantics for the job and whether the write path used idempotent identifiers or per-transaction markers. Flink has specific documentation about how it coordinates writes and commits; consult the Flink writes guide.
  • Inspect successive snapshots for repeated file additions that contain overlapping row ids. If present, the retry performed a second successful write.
  • If Flink was expected to write idempotently, look for misconfiguration such as missing transaction identifier propagation.

Resolution: Configure job-level idempotence where supported or add deduplication logic based on primary keys. Record the fix in the manifest and rerun the retry test.

Scenario 3: Schema evolution works in Spark but fails in Dremio

Symptoms: Spark can add an optional nested field and subsequent reads succeed. Dremio fails to read the new snapshots citing a schema parsing error.

  • Compare schema ids and field ids recorded by Spark's commit and Dremio's view of the table metadata. If Spark used a different field id mapping, Dremio's reader may not map fields as expected.
  • Check for engine-specific limitations documented in Dremio's schema evolution guide for nested data. Some engines have stricter rules for nested field renames or promotions.
  • Validate whether Dremio is using the same Iceberg client version and connector as Spark; if not, align versions or add a compatibility shim.

Resolution: If the issue is field id mismatch, perform a catalog-backed alter to harmonize field ids, or roll back and perform evolution through the same engine for a release gate. Document the exact change in the manifest and add a regression test.

Validation automation and where to stop

Turn the compatibility matrix into a release gate

FAILURE INJECTION AND RELEASE GATEObservecapture evidenceCompareuse a baselineDiagnosefind the boundaryActchange one variablemeasured resultunexpected changesmallest safe responsenew baseline
Failure injection and release gate.

A compatibility test becomes useful when it controls a real decision. A report that runs after deployment may explain an incident, but it does not prevent one. Put the contract in the path between an engine upgrade and production. The gate should evaluate the exact engine, connector, catalog, table format, and object-store combination that you plan to release. A passing result from a different connector version is supporting evidence, not a substitute.

I prefer three levels of evidence. The first is a fast pull-request suite with one small table and deterministic data. It catches changes to SQL generation, catalog configuration, and basic snapshot handling. The second is a nightly matrix that exercises every supported writer-reader pair. The third is a preproduction run against realistic data volume, partition counts, concurrent commits, and failure conditions. Each level answers a different question, so a green pull-request check should not be interpreted as proof that a busy production workload is safe.

The release artifact should identify more than a pass or fail. Store the engine image digest, connector artifact and checksum, catalog version, object-store endpoint class, table format version, test data seed, SQL files, expected snapshot ids, and the result of every assertion. Include the Iceberg metadata JSON, manifest lists, and query outputs when the run fails. These details turn an intermittent incompatibility into something an engineer can reproduce instead of a screenshot that starts an argument.

Separate writer certification from reader certification

Reading an Iceberg table and committing to it carry different risk. A reader can expose an unsupported type, delete file, or metadata feature without changing table state. A writer can publish a snapshot that every other engine must understand. Track those capabilities separately. An engine may be certified to read tables created elsewhere while its write path remains disabled. That is a useful operating mode during upgrades and a safer alternative to treating compatibility as a single yes-or-no property.

For each engine, define the table operations it owns. One organization may use Flink for streaming appends, Spark for maintenance, Dremio for interactive SQL and governed self-service analytics, and Trino for a specific federated workload. Your matrix should test those real ownership boundaries first. Testing every theoretical pairing can consume time while leaving the production path under-tested. Expand the matrix after the critical paths are reliable.

Reader certification should include current snapshots, time travel, tables with position deletes, tables with equality deletes, evolved schemas, evolved partition specs, and the data types your teams actually use. Writer certification should add commit conflicts, retries, partial failures, concurrent maintenance, and rollback behavior. If you enable Iceberg v3 features, create separate gates for each feature instead of assuming that support for the format version means complete behavioral parity.

Make failures easy to classify

A useful harness assigns every failure to a layer. Catalog failures include authentication, authorization, namespace handling, optimistic concurrency, and commit responses. Metadata failures cover schema ids, partition specs, manifests, snapshot references, and delete semantics. Storage failures include missing objects, inconsistent credentials, path encoding, and encryption configuration. Engine failures cover SQL behavior, planning, type mapping, caching, and connector-specific limitations. Infrastructure failures include timeouts, resource pressure, and unavailable services.

This classification matters because the same symptom can have several causes. A missing row may come from an uncommitted data file, an incorrect manifest, stale metadata in a reader, or a query predicate that one engine interprets differently. Capture evidence from every layer before the test environment is destroyed. Preserve catalog request logs, engine query ids, snapshot lineage, object listings, and the assertion that failed. The goal is not to collect everything forever. It is to retain enough context to reproduce the failure with the same versions and inputs.

Finally, give every exception an owner and an expiration date. If a known connector limitation requires skipping a test, record the affected engine versions, the exact operation, the business impact, and the condition that removes the exception. A permanent skip quietly changes the meaning of a green build. An explicit, time-bounded exception keeps the matrix honest and gives upgrade planning a concrete target.

You can automate most checks. Stop when you reach reproducible parity for the full contract you need: append, time travel, evolution, delete, merge, retry, and failure injection. If you are missing parity only for an experimental feature, keep that feature behind a gate and do not certify it for production across engines until you have per-engine fixes and the manifest shows passing runs for all engines involved.

Primary specification and engine write guides you must read and reference when implementing the tests:

Supporting Dremio resources (useful for planning cross-engine runs):

Keep learning

For a deeper treatment, download Apache Iceberg: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.

Use Dremio's Apache Iceberg guide as the evergreen reference for the wider architecture and terminology.

Sources

Practical recommendation

Start small and instrument aggressively. Run a minimal contract for append and time travel across all four engines with identical catalog and object store. Record the manifest with exact versions and connector artifacts, and iterate by adding schema evolution and delete tests. Gate any Iceberg v3 tests until you see consistent passing runs from every engine in your target production matrix. When a run passes the full contract for your chosen feature set, promote that manifest and run id as your cross-engine acceptance artifact.

Try Dremio Cloud free for 30 days

Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.