Healthy Iceberg monitoring combines client-side metrics reports with recurring metadata-table queries. Use ScanReport and CommitReport to capture planning, filtering, and write outcomes. Use the metadata tables to query file sizes, delete-file pressure, snapshot growth, and manifest shape. Together these signals let you set SLOs for snapshot freshness, commit reliability, planning latency, and maintenance debt, and they let you detect trends before query latency or storage bills spike.
Metrics reporters run in clients, which means the first place you will see writer and reader behavior is inside the process that opened the table. Metrics reports capture what happened during planning and commit, including rejected filters, rows scanned, and commit failures. Metadata tables, queried from a central engine, show the persistent physical state: file counts, file sizes, delete-file density, manifest counts, and snapshot chains. Both are necessary. Metrics show timing and intent, metadata shows accumulation and longterm cost.
Important technical cautions. Metrics reporters run in clients, and their names and payloads vary by implementation version. Verify the reporter implementation and payload schema for your Iceberg library version. Thresholds you set must follow workload baselines; do not copy absolute values from another environment without calibration.
Think of three signal layers.
On the implementation side, collect ScanReport and CommitReport via the reporter your client supports. Apache Iceberg documents a REST based reporter, and library-specific bindings exist for Spark and Flink. For a Java environment, RESTMetricsReporter is a shipped example that shows configuration properties and a simple HTTP sink. Use that as your starting point, but verify the reporter properties for your Iceberg version because names and payloads changed across releases. See the official metrics reporting guide for the version you run for exact property names.
These SLOs focus on ongoing observability, not one-time audits. They map directly to the signals above and to operational actions.
Iceberg exposes metadata tables that can be queried like normal tables. The Spark-specific queries in the Iceberg documentation show how to extract file and snapshot information. Use those queries as the basis for automated jobs. I recommend running these queries hourly for large clusters and every few minutes for critical low-latency tables.
Example health queries, adapted from the documented Spark queries. Verify syntax against your engine and Iceberg version before running.
<!-- wp:code --><code>-- 1. File-size distribution per table
SELECT
bucket(file_size, 0, 256*1024*1024, 10) AS size_bucket,
count(*) AS file_count,
sum(file_size) AS total_bytes
FROM table_name.files
GROUP BY size_bucket;
-- 2. Delete-file pressure: ratio of delete-files to data-files
SELECT
sum(case when file_type = 'data' then 1 else 0 end) AS data_files,
sum(case when file_type = 'delete' then 1 else 0 end) AS delete_files,
delete_files * 1.0 / nullif(data_files, 0) AS delete_to_data_ratio
FROM table_name.manifests_files;
-- 3. Snapshot growth rate
SELECT
snapshot_id,
committed_at,
row_count_delta,
file_count_delta
FROM table_name.snapshots
ORDER BY committed_at DESC
LIMIT 100;
-- 4. Manifest and manifest-list shape
SELECT
manifest_path,
added_snapshot_id,
deleted_data_files,
added_data_files,
deleted_files_count
FROM table_name.manifests;
</code><!-- /wp:code -->These example queries reference the metadata table names shown in the Apache Iceberg Spark queries guide. If you run queries from Dremio, treat the Dremio examples on querying internals as a practical how-to for connecting a SQL engine to those metadata tables. The Dremio blog post about querying internals explains how to reach internal metadata tables for analysis, which is useful when you want a centralized audit job across many tables.
RESTMetricsReporter is a common example implementation in Iceberg documentation. It shows the basic properties to configure an HTTP sink for metrics. Do not assume property names are identical across versions. Check the metrics reporting documentation for your Iceberg release. The official site documents a REST reporter and provides sample configs for various clients.
<!-- wp:code --><code># Example RESTMetricsReporter properties, verify these exact keys for your Iceberg version
metrics.reporter.rest.class=org.apache.iceberg.metrics.rest.RESTMetricsReporter
metrics.reporter.rest.endpoint.http://metrics-collector.internal.local/iceberg
metrics.reporter.rest.timeout.ms=5000
metrics.reporter.rest.batch.size=50
metrics.reporter.rest.include.scan-report=true
metrics.reporter.rest.include.commit-report=true
</code><!-- /wp:code -->What to verify after you add these properties.
Pick an automation cadence based on table activity. For very active tables pick 1 to 5 minute runs. For ad hoc analytics tables hourly is often enough. Below are practical queries you can run as scheduled jobs and push into Prometheus or your observability backend.
<!-- wp:code --><code>-- Small file ratio, as a percent
WITH stats AS (
SELECT
count(*) FILTER (WHERE file_size < 134217728) AS small_files,
count(*) AS total_files
FROM table_name.files
)
SELECT small_files * 100.0 / nullif(total_files, 0) AS small_file_percent FROM stats;
-- Average file size
SELECT avg(file_size) AS avg_file_bytes FROM table_name.files;
-- Delete-file density per partition (example partition column: ds)
SELECT
ds,
sum(case when file_type = 'delete' then 1 else 0 end) AS delete_files,
sum(case when file_type = 'data' then 1 else 0 end) AS data_files,
delete_files * 1.0 / nullif(data_files, 0) AS delete_to_data_ratio
FROM table_name.manifests_files
GROUP BY ds;
</code><!-- /wp:code -->Translate these results into timeseries by running them every minute and emitting the numeric result to Prometheus pushgateway, or by inserting results into a metrics table. The Spark queries documentation at Iceberg explains the metadata schema you will read, which helps you tailor these queries for your engine.
Below are Prometheus rule templates. Replace threshold numbers with baseline-driven values. These alerts assume you convert the query outputs into Prometheus metrics named ice_*. The exact metric names depend on how you instrument the scheduled jobs.
<!-- wp:code --><code># 1. Small file percent too high for the last 30m
- alert: IcebergHighSmallFileRatio
expr: avg_over_time(ice_table_small_file_percent[30m]) > 40
for: 15m
labels:
severity: page
annotations:
summary: "High small-file ratio for {{ $labels.table }}"
description: "Small files >40% for 30m, indicates compaction needed"
# 2. Delete-file pressure increase
- alert: IcebergHighDeleteToDataRatio
expr: max_over_time(ice_table_delete_to_data_ratio[1h]) > 0.5
for: 30m
labels:
severity: ticket
annotations:
summary: "High delete to data file ratio for {{ $labels.table }}"
description: "Delete files exceed 50% of data files, triggers VACUUM/expire snapshots"
# 3. Commit error spike as reported by clients
- alert: IcebergCommitFailureSpike
expr: increase(ice_commits_failed_total[15m]) > 5
for: 5m
labels:
severity: page
annotations:
summary: "Spike in commit failures for {{ $labels.table }}"
description: "Detects client-side commit failures from metrics reports"
</code><!-- /wp:code -->Prometheus rules assume you have converted ScanReport and CommitReport fields into metrics. When configuring the conversion, remember that reporter field names can change across Iceberg versions. Confirm the field names in your collector or the original reporter payloads. The official metrics reporting guide for your Iceberg release lists common payload fields and is the source of truth.
Alerts should map to an automated checklist and, where safe, automated remediation. For small-file and delete-file pressure, you can often run maintenance jobs automatically, but only after you verify safe conditions like cluster load and snapshot consistency.
Here are realistic failure modes, what emits them, and how to verify them.
When an alert pages, follow a short, repeatable triage sequence. Keep it simple so engineers can act under pressure.
After you deploy monitoring, treat these as the primary operating metrics. Measure them continuously and track trends.
Do not assume identical behavior across Iceberg versions. Metric names and payloads vary by implementation version. Always verify the reporter class and payload schema for the version you run by consulting the metrics reporting page for that version. For Spark the documented queries for metadata tables indicate column names and internal table names that may differ by engine and Iceberg release. I checked the references below on September 22, 2026 for guidance on these differences, but you must confirm the exact keys and table names in your environment.
Verification steps to run before relying on monitoring:
For practical how-tos when you are analyzing metadata tables from a SQL engine, see the Dremio blog post on Apache Iceberg metadata tables. That post shows concrete examples of reading files, manifests, and snapshots with SQL, which helps build the automated health queries mentioned above.
When deciding how to expire snapshots safely, read the Dremio piece on snapshot expiration. It explains risks and policies you should consider before automated snapshot pruning.
If you decide to delete orphan files or implement vacuuming after snapshot expiration, the Dremio orphan file cleanup post contains the operational steps and precautions for safely removing data files that are no longer referenced.
For general platform context about running Iceberg at scale with a modern SQL engine, the Dremio platform page provides guidance on integration patterns and performance considerations. Use it to map the monitoring signals into platform-level operational playbooks.
Do this sequence in a staging environment. Expect 3 to 6 hours for setup and a few hours of data collection. The goal is to validate end-to-end telemetry and demonstrate a full detection-to-remediation cycle.
Put simply, do not flip client metrics on for all writers at once. Start with a small, representative set of writers, validate payload format and payload volume, then expand. Below I give a concrete, staged rollout plan and a short test harness you can run in a controlled environment. The examples reference RESTMetricsReporter properties as documented by the Iceberg reporters pages. Verify the reporter property names in your client implementation and Iceberg version before applying them in production.
Stage 0: smoke test on a dev cluster. Configure one job or one Spark executor to emit metrics to a local HTTP sink you control. Confirm the reporter can reach the endpoint and the payload matches your parser expectations. Check for increased commit latency or unexpected exceptions in client logs.
Stage 1: 1 percent production traffic. Add the RESTMetricsReporter properties to a single task type that represents about 1 percent of your writes. This catches issues that only appear on production data while limiting blast radius. Collect traces and metrics for a few hours to cover peak writes.
Stage 2: 10 percent production traffic. Expand to 10 percent if Stage 1 is stable. Validate aggregated cardinality of metrics, monitor your metrics ingestion pipeline backing Prometheus and your metrics store for increased CPU and retention cost.
Stage 3: broad rollout. Gradually enable for all writers that produce new Iceberg table changes. Keep an eye on a few signals described below. If any signal degrades beyond a short lookup table, roll back to the prior stage.
Operational consequences to verify during the rollout:
All properties above may differ in name or availability by Iceberg or integration version. Check the Iceberg 1.6.0 documentation and the general metrics reporting reference for exact names and behavior. The test harness and staged rollout let you confirm actual behavior in your deployed client library version.
This harness uses a lightweight HTTP collector that logs each POST to disk, and a sample writer job that commits a small number of Iceberg writes. The goal is to validate payload shape, size, and error handling.
1) Start a collector. Any simple HTTP server works. For example, a minimal Python Flask app that writes requests to a file. Ensure you replace the import and run commands with what your environment allows. Confirm the server returns 2xx for valid requests.
2) Configure one writer to use the RESTMetricsReporter endpoint. Set batch size to 1 and interval to a small value while testing so you see immediate traffic. Capture the writer logs and the collector output simultaneously.
3) Execute a sequence of commits that mirror production write patterns, for example bulk loads, small updates, and delete operations. Compare the collector logs to the expected metric names from the Iceberg metrics reporting docs. Pay attention to which fields are present only on commit failures or compactions.
4) Induce a simulated outage of the collector and watch the writer behavior. Does the writer queue messages, retry, or drop metrics? Check for exceptions and increased latencies. This test reveals how tolerant the reporter is to temporary outages.
5) Validate size and cardinality. Save a few POST payloads and compute statistics: average payload size, number of unique label keys, and top label cardinalities. Use these numbers to estimate Prometheus label usage and store costs when scaling to all writers.
Running failure injection tests before production rollout is the difference between a recoverable incident and a production outage. Below are concrete fault injections you can run, the exact things to measure, and the expected signatures in logs, Prometheus, and metadata tables. These tests assume you can control the reporter endpoint and the writer process.
How to inject: configure the collector to sleep for N seconds on each incoming request. Test N values at 100 ms, 500 ms, and 2000 ms. Run a moderate write workload during each test.
What to measure: median and 95th percentile commit latency, error rate in writer logs, number of dropped metrics reported by the collector, and Prometheus scrape delays if you proxy metrics through an agent.
Expected signatures: if the reporter is synchronous, you will see commit latency increase proportionally to the added response time. If it is asynchronous with a bounded queue, you may see queue full errors in logs and dropped metrics. In Prometheus, missing metric samples or a sudden drop in metric volume are signs the reporter is failing to deliver.
How to inject: have the collector return HTTP 500 for 1 in 10 requests. Use deterministic patterns so you can correlate failed POSTs to specific writer commits.
What to measure: rate of retries, retry backoff behavior, whether retries block commit completion, and whether failed POSTs surface as exceptions. Check whether the client logs include stack traces or note suppressed telemetry errors only.
Expected signatures: repeated 5xx responses should increase retries and possibly exponential backoff. If retries are synchronous, expect longer commit times and potential backlog on the writer. If retries are asynchronous, you may see accumulating queued reports. Look for elevated memory use in writer processes when queueing grows.
How to inject: change the collector to validate expected JSON schema and return 400 on mismatch. Simulate a reporter version mismatch or a bug that omits fields to trigger 400 responses.
What to measure: whether the writer logs the payload error, whether it stops emitting metrics, and whether future payloads are accepted after correcting the schema. Also, check downstream parsers that may assume certain fields exist.
Expected signatures: schema errors usually indicate reporter/client-version drift. You will see 400 responses in the collector and possibly an immediate drop in metric samples or rapid error lines in writer logs. Fixing the reporter version or adjusting the collector to accept both formats usually resolves the issue.
Metrics reporter failures do not immediately change Iceberg metadata tables. But they do make troubleshooting harder. When you see missing metrics, use metadata table queries to detect symptoms that metrics would otherwise show, such as rapid growth in file count or large number of delete files. Run the health queries you already have scheduled and compare their results during the failure injection to the collector logs. If metadata tables show problems but metrics are absent, prioritize fixing the reporter ingestion path.
Alerts should map to human and automated actions. The following decision table maps combinations of Prometheus alert conditions, metadata table results, and recommended response. Adapt thresholds to your baseline and verify that the example Prometheus rules you included in the draft align with these actions.
Columns are intentionally concrete: alert, metadata verification query, immediate action, escalation, and automated remediation candidate.
Each action requires a verified rollback plan. For automated remediation, ensure you test and document the safety of any automated compaction or rewrite. Never auto-run a global rewrite without human approval and a tested rollback path.
Your draft includes Prometheus alert examples. Here are tuning suggestions tied to specific operational consequences, so they are actionable.
For each rule, include runbook links in the alert annotations. The runbook should point to the metadata queries that validate the condition, and to the staged rollback steps for reporter configuration.
Beyond firing alerts, add verification rules that help you triage. For instance, a rule that computes the ratio of metric series emitted per writer type helps locate a missing reporter deployment. Another rule that checks payload size averages can catch format regressions early.
Finally, ensure your alerting platform can silence alerts based on maintenance windows or deployment tags. If you perform a planned rollout of reporters, schedule silences in advance to avoid alert floods from expected metric volume changes.
Metrics without targets are noise. Below are practical operational metrics to measure after deployment, how to calculate them, and suggested starting targets. Tune these to your workload baseline and validate before making automated decisions.
Label each metric with the table name, environment tag, and writer role when possible. This lets you filter noise from background tasks and focus on critical paths.
Label cardinality is the biggest practical risk. If you discover the cardinality target is too low, consider sampling reporters, aggregating labels at job-level, or moving high-cardinality telemetry into a separate, high-capacity store.
Facts checked on September 22, 2026: Iceberg metrics reporting behavior and the documented reporter properties vary by client and version. Confirm property names with the Iceberg 1.6.0 metrics reporting documentation and the general metrics reporting reference linked in Sources.
These new sections are designed to slot into the draft before the FAQ. They add a concrete staged rollout, a quick test harness, failure injection tests, a decision table for alerts and automations, and a compact set of operational metrics with starting targets. They expand the practical material already present in your draft and give an operator actionable steps to validate and tune Iceberg metrics reporting at scale.
Emit ScanReport and CommitReport per query and per commit respectively. These reports are generated at the time of planning and commit. Do not batch them beyond what the reporter supports because that loses timing fidelity for debugging.
Reporters typically emit asynchronously or in a separate thread, but configuration matters. RESTMetricsReporter variants may batch or sync; check the reporter implementation and measure commit latency with and without the reporter enabled.
No. Metadata tables show the persistent state but not planning decisions or transient failures. CommitReport and ScanReport capture the operational timing and intent that metadata alone cannot show, such as failed commits or ineffective predicate pushdown.
Thresholds must follow workload baselines. Start by measuring current small_file_percent over 7 to 14 days, then set alert thresholds at a multiple of the normal variance, and tune down after a week of false positives.
Validate raw payloads after any upgrade. Keep a mapping table of field names per Iceberg version. When upgrading, run validation tests that exercise metrics emission before promoting the change to production.
See the Dremio blog post on Apache Iceberg metadata tables for concrete SQL examples. It shows how to read files, manifests, and snapshots from a SQL engine, which you will need to implement the scheduled health queries described above.
These guides cover adjacent implementation details that are outside this article's main scope.
For a deeper treatment, download Apache Iceberg: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.
To put the catalog and table-format ideas into practice, explore Dremio's Apache Iceberg platform.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
Apache Iceberg 1.12.0 expands v3 engine support, REST catalog operations, streaming reliability, and v4 groundwork. Learn how to upgrade safely.