A safe Apache Polaris upgrade treats the application, persistence schema, authentication behavior, and client compatibility as four separate gates. Do the reading, back up the persistence layer, test the exact multi-step upgrade path, apply required schema or index changes in a controlled maintenance window, canary the new version, and prove rollback remains practical before expanding to the whole fleet. This article gives a versioned operations runbook, worked examples, and concrete checks to make staged upgrades repeatable across clusters such as the canary cluster, the regional production cluster, and the recovery cluster.
Polaris upgrades can touch several independent systems. The control plane code changes, the backing persistence store (for example Postgres), and authorization or token behavior can each change compatibility boundaries. Some of these changes are rolling, others are not. Notably, persistence-type migrations are not rolling upgrades. If a release requires a data format migration, you cannot simply restart nodes one at a time and expect a transparent transition. The wrong assumption about which changes are rolling is the most common operational mistake I have seen.
This runbook is practical. It separates decision points: when to apply schema changes, when to canary, what to test for rollback, and what to measure after deployment. I reference the Polaris project evolution notes and the Helm chart production guidance where appropriate. Where the Polaris docs document behavior, I quote the location and explain what to verify in your environment before you start. Facts that depend on project documentation are checked on October 27, 2026.
Execute upgrades in phases. This list is an operational sequence, not a script. Each phase ends with explicit acceptance criteria that must be met before proceeding.
Each of these items is expanded below, with concrete examples, commands to run where the upstream docs provide them, and failure modes to watch for.
Before you change anything, draw a dependency map for your stack. Polaris application versions depend on schema versions, Helm chart values, and client compatibility. Persistence migrations can require exclusive access. Authentication/token behavior changes can break API clients even when nodes are otherwise healthy. Here is a concise checklist to capture dependencies:
Map each dependency to clusters. I maintain a simple inventory for each cluster, for example the canary cluster is a single-region fleet, the regional production cluster is multi-az, and the recovery cluster is a DR standby. That classification drives how much downtime I accept for schema migration and whether I coordinate cross-cluster cutover.
Start with a worksheet. Below is a worked example you can copy into a spreadsheet. The worksheet makes decisions explicit: can the client remain on the old token flow, must tokens be reissued, or do you need a migration step that updates stored tokens?
Worked example, fields and sample entries:
Use these fields to decide whether you can canary or must perform a full maintenance window. For the canary cluster in this example, the presence of a persistence-type migration forces a short maintenance window even though code changes would otherwise be rolling.
Backups have to be fast, verified, and restore-tested. For Postgres, take a base backup plus WAL segments to enable point-in-time recovery to the exact pre-migration moment. If you use a managed Postgres, export logical dumps in addition to provider snapshots. Save connection strings, superuser credentials, and a restore playbook as part of the runbook.
Decide whether the schema change is rolling or non-rolling. If the migration touches on-disk layout or requires a background compaction that rewrites primary keys, treat it as persistence-type. Persistence-type migrations are not rolling upgrades. That means you must plan a maintenance window, place the cluster into reduced traffic or read-only mode if you can, and run the migration as a coordinated single-step change.
If the project documentation shows a migration script or SQL, validate it in a staging environment with production-sized data. The Polaris in-development evolution notes and the Helm chart production guidance describe migration expectations and recommended operators. Where the docs do not include a script for your exact path, record the SQL you will run and test the reverse path.
When you have a schema change, you must orchestrate application rolls and SQL changes carefully. Here is a safe sequence that handles rolling-compatible schema changes and non-rolling persistence-type migrations. I list explicit acceptance criteria after each step.
Remember, schema changes can limit rollback. For example, a migration that drops columns or rewrites primary keys may make the prior application binary incompatible with the database. Identify a cutoff step where rollback is no longer possible and schedule that step inside your maintenance window with senior approval. Record the exact migration statements and the expected duration.
When using Helm to deploy Polaris, set explicit canary values in a values file, and keep the defaults immutable in your chart repository. Below is an example canary values file you can adapt. Use this pattern to run a single canary deployment per cluster while leaving other replicas on the old release.
<!--
values-canary.yaml
-->
replicaCount: 1
image:
tag: "1.8.0"
pullPolicy: IfNotPresent
service:
annotations:
rollout: canary
resources:
requests:
cpu: "500m"
memory: "1Gi"
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 10
livenessProbe:
initialDelaySeconds: 60
periodSeconds: 20
# enable only the features you are validating
featureFlags:
newTokenFlow: true
database:
migration:
autoApply: false
-->
Notes about this example: set database.migration.autoApply to false if you plan to apply schema changes manually. The Polaris Helm chart production guidance documents chart values and recommended production defaults; use it to confirm which keys control migration and probes before you run a canary. See the Polaris chart production guidance and the 1.8.0 Helm chart release notes to confirm the exact keys available in the chart for your target version.
Rolling upgrades work only when the older and newer versions are mutually compatible for a time window. Define that window explicitly. For example, if your application can accept both old and new token formats for 48 hours, your compatibility window is 48 hours. After that, you must stop accepting the old format or complete the migration of all clients.
Design the rolling window with three parameters: grace period, canary observation time, and per-wave stabilization time. Reasonable defaults I use: grace period 48 hours for auth and token exchanges when tokens have long TTLs, canary observation 30 to 60 minutes under load, and per-wave stabilization 15 to 30 minutes for stateless nodes. Adjust these when stateful behaviors or background jobs are involved. When Polaris documentation specifically calls out token exchange changes, adjust your grace period down or require token re-issuance for clients that cannot handle the new flow.
Measure compatibility by tracking error counts for authentication endpoints, the fraction of requests that trigger token exchange logic, and the number of clients upgraded. For clusters with external client fleets, instrument a migration progress dashboard to show percent upgraded by client version. That dashboard is a key input to the decision to proceed beyond canary.
Rollback is not a single operation. It is a set of decisions you make before you touch the cluster. First, classify the upgrade by rollback risk:
Prepare an explicit rollback playbook and run it in a staging environment. The playbook should include acceptance tests. Below is a worked example of a rollback acceptance test you can perform before upgrading production. Keep a copy of it in your runbook.
1) In staging, deploy the old release and snapshot data.
2) Deploy the new release and apply the exact schema migration steps planned for production.
3) Run a suite of writes and reads that exercise new fields and token flows, capturing sample tokens and response shapes.
4) Execute rollback by deploying the old binaries and running any down-migration SQL you have prepared. If the migration is irreversible, restore the database from the snapshot and bring the old binaries back up.
5) Acceptance criteria: all previously captured queries and samples succeed, client SDKs can authenticate with restored tokens or reissued tokens per your client plan, and no data loss beyond known acceptable window.
Label the point where rollback requires database restore. Make that state visible in your runbook and have stakeholders sign off before you proceed past it in production. Remember that schema changes can limit rollback, and that a restore may take longer than a pod roll and require additional coordination with networking, DNS, or upstream clients.
Use this decision tree during an incident. I include clear triggers that reflect real-world signals operators can see in dashboards.
<!--
Start
|-- Are there schema changes that rewrote or dropped data? (Yes/No)
|-- No --> Can we roll pods back? (Yes) --> Rollback pods in waves, monitor for stability.
|-- No --> Perform coordinated restart, check Helm chart values.
|-- Yes --> Do we have verified point-in-time backup? (Yes/No)
|-- Yes --> Restore DB snapshot to isolation environment, verify, then promote.
|-- No --> Emergency freeze writes, notify stakeholders, prepare for manual reconciliation.
-->
Put this decision tree on the incident response whiteboard and make sure the on-call engineers can access the backups and have credentials required for restore. For clusters that serve as DR or standby such as the recovery cluster, rehearse the restore and promotion steps so you can cut over quickly if needed.
After a successful rollout, measure both correctness and operational impact. Track these metrics continuously for at least two full compatibility windows, as defined earlier.
Instrument these metrics in your monitoring system with alerts that are meaningful. An alert that fires for a short blip during a rolling pod replacement is noise. Alerts should tie to sustained error rates or violation of SLA thresholds over a defined window.
Use the Polaris project pages for the authoritative upgrade and chart details. The in-development evolution notes list planned migrations and behavior changes. The Polaris Helm chart production guide documents recommended values and operator guidance for production deployments. The Polaris 1.8.0 release page includes specific notes about token behavior changes documented by the project. I also keep a short operator guide that explains running Polaris on Kubernetes, which is useful for cluster-specific troubleshooting and connecting Helm chart features to real deployments. The following Dremio resources help:
For hands-on cluster examples and additional architecture context, these Dremio posts are practical: the walkthrough on running Polaris in production with Kubernetes, Helm, and Postgres includes operational tips I used when managing the canary cluster and the regional production cluster. The architecture explainer is useful for mapping component responsibilities. The Polaris primer helps non-operators understand which capabilities are affected by each upgrade step.
Do not assume every chart key or migration behavior is the same across releases. The Polaris Helm chart evolved between the in-development guidance and the 1.8.0 chart release; consult the chart docs for your exact release. If the supplied Polaris project documentation does not include a ready-made down-migration, you must design and test the reverse SQL path, or accept that rollback requires a database restore. Do not start an upgrade without a tested restore and rollback rehearsal.
When scheduling upgrades across multiple clusters, stagger by role. Upgrade a canary cluster first, then a regional production cluster such as the canary cluster, then a multi-region like the regional production cluster, and leave a DR cluster such as the recovery cluster for final validation. That order preserves recovery options and minimizes blast radius.
Start this step with the pre-upgrade compatibility worksheet filled from the draft. The goal here is to convert that worksheet into an executable checklist you can run against a staging cluster that mirrors production. Run these commands and checks in order, and mark pass or fail for each. Do not proceed past a failed check until you either fix the issue or accept the increased risk.
Target versions matter. The Polaris project published the 1.8.0 release and Helm chart details in the official release pages and the in-development Helm production guidance. When I describe behavior, verify it against those documented pages for Polaris 1.8.0 and the production chart sources. Facts checked on October 27, 2026.
Use a staging cluster that mirrors these dimensions: same persistence backend type (etcd, PostgreSQL, or MySQL as used in your production), same network topology for tokens, identical ingress behavior, and the same client versions. If you cannot match all of those, explicitly record the mismatch on the worksheet. Missing a persistence match invalidates any migration run planning that assumes non-rolling behavior.
Below I list concrete checks you can add to the worksheet, with example commands where Polaris documentation provides the resource. Replace placeholders such as STAGING and NAMESPACE before running. Do not invent cluster keys; consult your environment for exact resource names.
When each item passes, mark it green. If anything fails, add a short remediation action and a timebox for the fix. Do not skip the persistence migration dry run. Persistence-type migrations are not rolling upgrades, and the worksheet must record the exact migration steps and their irreversible effects.
The existing draft has a worked example. Here I add a mutation table that maps a small set of Helm values to observable behavior, and a concrete plan for changing values during canary waves. Use the Polaris Helm chart documented in the official release pages for exact value names. Verify keys against your chart version before applying them.
Label the following as your baseline canary wave values. The names below are illustrative; replace them with the actual keys from the Polaris production chart documentation for 1.8.0.
Use a mutation table like the one below in your runbook. Each row is a single, scoped Helm value change. For each change, record the expected impact, the validation command or test, and the rollback action if the canary fails.
Practical sequencing: apply one Helm value mutation at a time. Wait for the canary validation checklist item that maps to that mutation to complete before the next Helm change. This isolates failures so you know which value caused the regression. If a mutation must touch persistence behavior, treat it as a migration and move to a non-rolling, maintenance-window plan.
The draft has a rollback acceptance test. Expand that into a reproducible scenario you can run on staging to validate your rollback playbook including Helm and persistence paths. This test proves your rollback decision tree and acceptance criteria are actionable.
Start with a live staging release that mirrors the production Helm values and persistence copy. Use two release tags so you can roll forward and then roll back with Helm. Label them vOLD and vNEW. Ensure you have a snapshot of the persistence taken immediately before the vNEW migration if persistence changes are involved. If you cannot snapshot the exact production storage engine the same way, record that limitation; it changes rollback feasibility.
Inject these faults one at a time and run the acceptance checks after each. Each step is intended to exercise a different branch of the rollback decision tree.
Automate these checks in your CI/CD pipeline so they are quick to run before a production push. For each rollback scenario, the acceptance test must include:
When a rollback acceptance test fails, record the exact failure with logs, timestamps, and the Helm commands used. That information shortens live incident debug time. The rollback test must prove that your documented Helm rollback commands, combined with persistence snapshots when applicable, return the system to a known good state or halt with a clear remediation plan.
Beyond the general metrics in the draft, here are concrete signals to collect, how to calculate them, and operational thresholds to use as part of the canary acceptance criteria. Record baseline values on the pre-upgrade worksheet so you can compare.
Set alerts for each threshold. Make them distinct from generic CPU or memory alerts so operators know a canary failure is due to the upgrade, not unrelated infrastructure noise. For every triggered alert, capture the dump of the pre-upgrade worksheet entry that corresponds to the metric, so the team has immediate context.
Do these tests in staging before attempting production, and record results back into the worksheet. The goal is to exercise each branch of your rollback decision tree with precise actions and acceptance criteria. Below is a concise decision table you can paste into your runbook.
Each table row should contain the exact Helm commands you will run, the persistence snapshot location, the people to notify, and the maximum allowed time for each step. These details convert theory into a playbook you can follow under incident pressure.
Finally, update your pre-upgrade worksheet, Helm canary values, and rollback acceptance test documents after every rehearsal. That is how the runbook stays accurate and how you avoid discovering a token exchange behavioral mismatch, a missing DB privilege, or an irreversible schema change during a live upgrade.
Not always, but treat them as non-rolling until proven otherwise. If the migration rewrites on-disk formats, primary keys, or requires exclusive table locks, you cannot rely on pod-by-pod restarts. Verify the exact SQL and test it against production-sized data.
Polaris 1.8 changed token exchange behavior as documented on the 1.8.0 release page. That can shorten the compatibility window or require token reissuance for some clients. Test authentication flows end-to-end in canary and verify how long old tokens remain valid.
You can automate pod rollbacks for code-only regressions. Helm rollbacks revert release metadata and can trigger pod restarts. They do not revert irreversible database migrations. For migrations that altered data, your rollback is a database restore plus redeploying the old chart values.
Keep canary size small enough to limit blast radius, but large enough to surface relevant behavior under load. For control-plane services I typically run one replica for functional canaries and 5 to 10 percent of traffic for load validation. Use the Helm canary values example earlier to pin replicaCount and resource requests.
Observe for at least the length of your longest usual failure mode. Practical defaults: 30 to 60 minutes for API correctness, and 48 hours for auth/token compatibility when tokens are long lived. Extend the window if background jobs or long-running queries might expose issues.
Start with the Dremio walkthrough on running Polaris in production for hands-on operational tips and backup guidance. Use the architecture explainer to map components, and the primer to educate stakeholders about which services will be affected during the upgrade.
These guides cover adjacent implementation details that are outside this article's main scope.
For a deeper treatment, download Apache Polaris: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.
To put the catalog and table-format ideas into practice, explore Dremio Open Catalog.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
Iceberg keeps per-file bounds automatically but table-level NDV statistics only when you compute them. Why that decides join order, and how to keep them fresh.
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.