Server-side scan planning moves Iceberg table scan work from the client to the REST catalog service, so the catalog reads manifests and creates file-scan tasks near the metadata instead of shipping all manifest lists and entries to the client. A client first discovers whether the catalog supports this feature, then submits a scan request and receives either an immediate task batch or an asynchronous plan identifier to poll for task batches. The approach reduces metadata transfer and client CPU on large tables, but it converts planning load, latency variability, and observability requirements into catalog responsibilities. This article explains the protocol behavior, shows exact example requests and responses, and gives an operator playbook for capacity, failure modes, testing, and rollout.
Large Iceberg tables can have millions of manifests and billions of entries. In classic client-side planning the engine downloads manifest lists and manifests, parses every file metadata record, filters, and emits file-scan tasks. That moves network, CPU, and memory costs to the engine. Server-side planning keeps those costs where the metadata lives, which can drastically reduce client-side bandwidth and memory for wide, partitioned tables. But there are trade-offs. Planning capacity now becomes an operational SLO for your catalog, and you must treat the REST endpoints as optional: clients must use capability discovery to decide whether to switch to server-side planning. The protocol adds asynchronous states, request cancellation, and batched task delivery that change how engines schedule work and how operators observe failures.
At a protocol level the flow is: capability discovery, prepare a scan request, submit it to the catalog, then either accept immediate tasks or poll an asynchronous plan. Polling returns zero or more "task batches" that contain drive-level file tasks. Engines can cancel a plan if they change query shape or a user aborts the job. The client must not assume any endpoint exists unless capability discovery reported it; the endpoints are optional and version-dependent. The catalog may impose size limits on returned batches and time limits for planning; these are implementation details the client must be ready to handle.
The practical difference is which process reads the manifests and applies split-generation logic. With client-side planning the engine downloads each manifest and creates tasks; with server-side planning the catalog returns tasks already computed. That changes failure surfaces and observability. On client-side planning a slow network link between engine and storage shows up as slow manifest fetches on the engine. With server-side planning a slow or overloaded catalog shows up as delayed plan completion or throttled batch delivery.
Example consequences:
For an implementation guide to REST catalog basics and how engines talk to the catalog API, read Dremio s writeup on the Iceberg REST catalog, which shows request patterns and operational details specific to Dremio s environment. That post also links to common failures when the catalog is misconfigured.
Iceberg added a REST catalog protocol that includes optional endpoints for server-side planning. Clients must call a capability endpoint first. If the catalog advertises planning endpoints, the client can proceed. If not, the client falls back to client-side planning or to another catalog such as an S3 or Hive catalog. The endpoints are optional by design, and capability discovery must drive client behavior.
The protocol provides both synchronous and asynchronous variants. A synchronous submission may return an immediate batch set if planning completed quickly. The asynchronous variant returns a plan identifier and requires the client to poll for batches. The client uses a cancel endpoint to stop a plan if the query is aborted or the scheduler wants to replan.
Below are three worked examples you can paste into a test harness. They are taken from the OpenAPI spec and the nightly protocol documentation. The exact JSON may vary by Iceberg version and the catalog server implementation. Verify these fields in your catalog by consulting the catalog s OpenAPI definition and runtime responses.
<!-- wp:code -->
{
"catalogType": "rest",
"rest": {
"supportsScanPlanning": true,
"scanPlanning": {
"submitPath": "/api/v1/scan/submit",
"pollPath": "/api/v1/scan/poll/{planId}",
"cancelPath": "/api/v1/scan/cancel/{planId}",
"maxBatchSize": 1000
}
}
}
<!-- /wp:code -->Notes: the real response keys and nesting depend on the catalog OpenAPI version. Check the catalog s OpenAPI YAML for exact names and whether the scanPlanning object is present. Do not assume presence; the endpoints are optional. The OpenAPI definition in the upstream Iceberg repository documents the expected schema for the REST catalog, and the nightly protocol page documents behavioral semantics. See both sources in the Sources section.
<!-- wp:code -->
POST /api/v1/scan/submit
Content-Type: application/json
{
"tableIdentifier": {
"catalog": "my-rest",
"namespace": ["prod","events"],
"name": "clicks"
},
"projection": ["user_id","event_time","page"],
"filter": "event_time >= '2026-01-01'",
"splitSizeBytes": 536870912,
"maxTasksPerBatch": 500
}
<!-- /wp:code -->Possible responses:
<!-- wp:code -->
GET /api/v1/scan/poll/abcd-1234
Response 200
{
"planId": "abcd-1234",
"state": "IN_PROGRESS",
"batches": [
{
"batchId": 1,
"isLastBatch": false,
"tasks": [
{"filePath": "s3://bucket/table/part-0001.parquet","start":0,"length":1048576},
{"filePath": "s3://bucket/table/part-0002.parquet","start":0,"length":2097152}
]
}
]
}
DELETE /api/v1/scan/cancel/abcd-1234
Response 204
<!-- /wp:code -->Notes on polling: polls may return empty batches to indicate planning is still ongoing, or partial batches if the catalog throttles responses. Clients must handle idempotent retries for poll calls and be prepared for duplicates. The protocol should include batch identifiers; use them to deduplicate at the engine and to commit progress. The ability to cancel must be honored within an SLO; a canceled plan frees resources and may return a partial set of tasks already emitted.
Designing the state machine for async planning matters for correctness and for operator observability. A minimal state machine includes: ACCEPTED, IN_PROGRESS, PARTIAL, COMPLETED, FAILED, CANCELLED, and EXPIRED. Plan identifiers must be unique and have a TTL so the catalog can retire old state. The catalog must persist plan progress in a way that tolerates process restarts; otherwise a long-running plan will disappear when a planner pod restarts and clients will see failures or indefinite waits.
State transitions and operational consequences:
Instrumentation to expose: plan age, time in each state, CPU and I/O used per plan, number of batches emitted, and bytes of manifest scanned. These metrics let you define catalog SLOs and build alerts for overload. If your catalog is fronted by an API gateway, maintain tracing across the gateway and your planner components so you can attribute latency to the right microservice or node.
Batched delivery is the unit of work the catalog hands to the engine. Batches should be small enough that a single engine worker can accept and launch tasks without running out of memory, but large enough to amortize round trips. A typical batch size is 100 to 1000 tasks depending on task size. The OpenAPI spec contains an example and fields for batch items and is the authoritative reference for exact payloads, so validate how your catalog implements batching by inspecting its OpenAPI definition.
Engine-side responsibilities:
Server-side planning moves manifest scanning cost to the catalog cluster. Treat planning capacity like any other cluster resource: measure, set SLOs, and provision for peak concurrency. Planning capacity is a first-class catalog SLO. If you do not set it explicitly, a spike in large queries can exhaust the catalog and stall other users.
Key capacity planning inputs:
Failure domain design: planners should be stateless workers with a shared durable store for plan state. That reduces the blast radius of a single planner crash. Store plan metadata and progress in a fault-tolerant store that survives restarts, such as a database or durable object storage. If you keep state only in local memory, a planner pod restart can make in-flight plans vanish and cause client timeouts or errors. The state machine section above lists EXPIRED for abandoned plans; implement TTLs so state does not accumulate indefinitely.
Operational signposts and alerts to create:
Do not flip a catalog to server-side planning without testing it against production scale. The following sequence is what I run in production rollouts. It moves from low risk to high risk and exercises both functionality and operational controls. Each step contains measurable pass/fail criteria.
During load tests, pay attention to tail latency. Planning often exhibits long tails when a single manifest is on a slow storage medium or when planners contend on a particular metadata shard. Build timeouts and backoff in the client to avoid cascading failures and to keep cancel rates manageable.
Expect these failure modes and design remediation for each.
When troubleshooting, correlate planIds with API gateway logs, planner pod logs, and object-storage access logs. For example, a plan that stalls on a particular manifest will show repeated storage GETs for that manifest in object-store logs. Use that to isolate slow or failing storage backends.
Measure the following and set alert thresholds tied to your SLOs:
Label any metrics verified on September 22, 2026 where freshness matters, for example counts and behavior tied to protocol versions that changed after that date. The Iceberg REST protocol documentation and the OpenAPI definition are authoritative; check them against your catalog implementation when you measure these metrics.
Dremio has several posts that are useful for operators integrating server-side planning. The Dremio blog post on the Iceberg REST catalog explains what the REST catalog is and how to use it, which helps with capability discovery and example requests. The Dremio engineering post about Apache Polaris shows how engine-to-catalog APIs are used in practice and maps to patterns you will see in REST catalog interactions. For debugging metadata and verifying that the planner sees the same manifest state as your engine, Dremio s guide to Iceberg metadata tables helps you examine internal table state without scanning every manifest. Finally, Dremio s Open Catalog platform page documents how to configure catalog endpoints and security settings in Dremio, which you will need when wiring your engine to a REST catalog.
I linked each Dremio page in the relevant context so you can jump directly to the part that helps your immediate task: capability discovery, engine-to-catalog interaction patterns, metadata inspection, or platform configuration.
Do not assume every REST catalog implements the same set of fields, sizes, or guarantees. The endpoints are optional. Verify these items before you rely on server-side planning:
Check the official Iceberg REST protocol documentation and the project's OpenAPI spec when you audit these behaviors. The links in the Sources section point to the nightly protocol documentation, the 1.11 release blog that describes the feature set history, and the repository OpenAPI definition. Verify fields and behaviors against the exact Iceberg version you deploy.
This section gives a concrete, defensive client implementation pattern that you can drop into an engine or connector. It assumes your client has already fetched the catalog capabilities response and seen support for server-side planning. The examples build on the worked capability, submit, and poll snippets in the draft, and add retry, backoff, and guarding logic you should test before enabling this path in production.
Start by splitting responsibilities. Treat the catalog like a stateful planner and the engine as a stateless executor. The client code has these responsibilities: (1) discover capabilities and capacity window, (2) submit a scan request, (3) poll for tasks, and (4) hand tasks to the engine worker pool. The following pseudo-code highlights the key decision points and error handling. Replace placeholders with your HTTP client, JSON parsing, and task-execution code.
// Pseudocode: defensive server-side planning client
// 1. Discover capabilities
cap = http.get("/capabilities")
if not cap.supportsServerSidePlanning:
// fallback to client-side planning
planLocally()
return
// 2. Check advertised concurrency and backlog window
maxConcurrent = cap.maxConcurrentTasks or 1
backlogWindow = cap.backlogWindowSeconds or 60
// 3. Build submit request
submitReq = buildSubmitRequest(table, filters, projection, splitHints)
resp = http.post("/submit", submitReq)
if resp.status == 503:
// catalog busy, back off and retry locally after N attempts
maybeRetryOrFallback()
elif resp.status != 202:
fail("unexpected submit response")
scanId = resp.body.scanId
// 4. Poll loop with bounded concurrency and jitter
openTasks = 0
while not scanComplete:
if openTasks < maxConcurrent:
pollResp = http.get("/poll", {scanId: scanId})
if pollResp.status == 200:
tasks = pollResp.body.tasks
for t in tasks:
openTasks += 1
submitToEngine(t, onComplete = lambda: onTaskDone())
elif pollResp.status == 204:
// no tasks now, sleep with backoff
sleepWithJitter()
elif pollResp.status == 410:
// server indicates scan canceled or expired
handleScanCanceled()
break
else:
// throttle polling to avoid overloading the catalog
sleepShort()
function onTaskDone():
openTasks -= 1
// 5. Cancel path: if engine cancels, send cancel
if engineRequestsCancel:
http.post("/cancel", {scanId: scanId})
Notes on the code above. Respect the catalog advertised maxConcurrentTasks. The catalog may include a backlog window value, which must shape client retry and polling frequency. Treat 503 responses from submit as a transient overload, but do not retry forever. The catalog is an operator controlled component, not an always-available library. Implement a fallback to client-side planning after a configured number of submit failures or a per-query timeout.
Before flipping a flag in production, run a small suite of failure injection tests against a staging catalog. These tests exercise the exact failure modes operators see when capacity is tight, networking glitches occur, or the catalog restarts. Use the worked capability, submit, and poll examples as the baseline message shapes your test harness will send and validate. Below are four concrete tests to run, what to expect, and what your client must do.
Test: Configure the catalog to return 503 for submits when an artificial backlog threshold is reached. Send a mix of 100 concurrent client submit requests for moderately sized scans. Observe how many clients get 503s and how many proceed to fallback planning.
Expected client behavior: After a small number of retries with exponential backoff and jitter, clients should either fall back to client-side planning or surface a clear error to the caller. Do not queue unlimited retries on the client, that amplifies overload. Track submit failure ratio as an SLI for the catalog.
Test: Make the catalog return partial task lists on poll, and sometimes repeat tasks across polls. Introduce a short network partition so the client misses an ACK and retries. Your engine must tolerate duplicate tasks. Verify retries are idempotent or detect and drop duplicates at execution time.
Expected client behavior: De-duplicate by task id before scheduling or make task execution idempotent. If tasks include sequence numbers, validate order where the catalog documents ordering guarantees. If ordering is not guaranteed by the protocol version you use, clients must not assume it.
Test: Submit a scan, start receiving tasks, then send cancel. Simultaneously crash half the workers handling tasks. Confirm the catalog accepts cancel and that any tasks in work-in-progress are handled according to your engine semantics. The catalog may continue returning tasks until it observes cancellation, depending on implementation details and capability version.
Expected client behavior: After cancel is acknowledged, stop polling and drain in-flight tasks using a short graceful timeout. If workers crash, ensure a higher-level query coordinator reconciles partial results and reports a consistent failure to users. Record cancel latency from client request to catalog acknowledgement.
Test: While a long-running scan is planned server-side, restart the catalog process or clear its in-memory state. Confirm whether the scan id remains valid and whether tasks continue to be returned post-restart. This checks durability requirements the draft lists under what must be durable.
Expected client behavior: If the catalog documents durable planning state for the release you run, clients should resume normally. If not, clients must detect missing scan ids and either resubmit the request or fall back. The test will show which behavior your catalog supports. Record a metric for plan-state-loss events and include it in the catalog SLO review.
This table is a short decision matrix engines should apply at runtime. It expands on capability discovery and planning capacity concepts in the draft. The client must inspect the capabilities response, operator SLOs, and local resource availability before choosing server-side planning.
// Runtime decision checks, evaluated in order
1. Capabilities check:
if cap.supportsServerSidePlanning is false: plan locally
2. Version and guarantees:
if cap.protocolVersion < minimumSupportedVersion: plan locally
3. Catalog capacity and backlog:
if cap.currentBacklog > cap.backlogWindow * safetyFactor: plan locally
4. Engine resource pressure:
if engine.cpuUtil > engineCpuThreshold or executorQueueDepth > X: plan locally
5. Query SLO alignment:
if queryDeadline < expectedServerPlanningLat: plan locally
6. Operator policy override:
if operatorForcesLocalPlanningForSomeTables: plan locally
Else: use server-side planning
Explain the checks. The capabilities response is authoritative for whether planning endpoints exist and what concurrency the catalog supports. Check protocol version if your client relies on specific guarantees like task ordering or durable plan state. Some catalogs may advertise a real-time backlog value. When available, compare that to the backlog window to avoid agreeing to long waits that violate your query SLOs.
Engine resource pressure matters. Server-side planning reduces client CPU and memory usage, but increases the catalog load and shifts some latency into the catalog. If your engine is already overloaded, local planning might be impossible, so you must weigh which component has more spare capacity. Finally, include an operator override. Operators may want to restrict server-side planning for particular tables, workloads, or tenant groups.
On the question of optional endpoints, remember the endpoints are optional server implementations may omit submit, poll, or cancel. Your client must not assume presence beyond the capabilities response. If an endpoint is absent, fall back to local planning for that query type.
Server-side planning makes the catalog an active part of the query path, so the catalog operator must treat planning capacity as part of the catalog SLO. Below are concrete metrics to emit from the catalog and to capture on the client. These support incident triage and capacity planning.
Catalog-side metrics to expose
Client-side metrics to capture
How to shape SLOs. Make planning capacity an explicit SLO for the catalog. For example, an operator could set a target that 99 percent of submit requests that are accepted must receive at least one task within 5 seconds, for scans under a size threshold. This example threshold depends on your workload and must be validated. When the catalog advertises concurrency, include the advertised limits in a capacity plan with headroom for spikes. If the catalog exposes current backlog, track backlog percentiles and sustain a margin so the backlog does not exceed the backlogWindow the catalog advertises.
Finally, correlate catalog metrics with engine-side SLIs. Track end-to-end latency from query submit to first row returned, and tag traces with whether server-side planning was used. That gives an apples-to-apples comparison for operational decisions.
Clients must call the catalog capability discovery endpoint and inspect the response for a scanPlanning object or equivalent fields. The endpoints are optional, so capability discovery must drive client behavior.
Treat planning capacity as a catalog SLO. If the catalog becomes overloaded, it should respond with throttling status codes such as 429 or with increased planning latency. Clients should implement backoff and fall back to client-side planning or retry later, depending on policy.
Do not assume strict ordering unless the spec and your catalog implementation guarantee it. Polls may return partial or out-of-order batches. Clients should deduplicate using batch and task identifiers and handle missing batches by re-polling until a final isLastBatch is observed or the plan expires.
At minimum, plan state and at-least-once records of emitted batches should be durable across planner restarts. If state is only in memory, a restart will drop plans and force clients to re-submit. Durable storage can be a database, object storage, or any fault-tolerant store documented by your catalog implementation.
Use task identifiers plus batch identifiers to deduplicate. If those are not present, deduplicate by file path and byte ranges. Ensure idempotent reads at the engine level to avoid double processing side effects.
Start with plan submission rate, planning latency p95/p99, planner CPU and I/O, batch emission rate, and cancel rate. These metrics will quickly tell you if planning causes unexpected resource pressure or if client timeouts need tuning.
These guides cover adjacent implementation details that are outside this article's main scope.
For a deeper treatment, download Apache Iceberg: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.
To put the catalog and table-format ideas into practice, explore Dremio's Apache Iceberg platform.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]