Apache Polaris is a catalog service for Apache Iceberg. It answers the question “which files make up this table right now, and are you allowed to touch them?” Its architecture has four moving parts: an entity model that nests catalogs, namespaces and tables inside a realm, a persistence layer that stores those entities and their grants, a REST service that speaks the Iceberg REST Catalog API, and a credential broker that hands engines short-lived, path-scoped access to object storage.
Most architecture diagrams put the catalog in a small box off to the side, wired to everything with thin grey lines. That box is doing more work than the diagram suggests. Every read, every write, every schema change and every permission check in an Iceberg lakehouse goes through it. If it is slow, your query planning is slow. If it is wrong, you get partial reads. If it is compromised, so is your storage.
Apache Polaris reached 1.7.0 and graduated to a Top-Level Project at the Apache Software Foundation. Dremio co-created it and donated it to the ASF, which is worth saying out loud because it shapes what follows: the design decisions in Polaris were made in public, and you can read them.
This post walks through what Polaris actually does when an engine asks for a table, and why the answer involves a credential broker rather than a lookup table.
Polaris has more moving parts than a catalog usually gets credit for. This walks through the entity model and the naming rules it enforces, how realms isolate tenants at the storage layer, the full request path from loadTable to a scoped storage credential, the exact vended credential properties for AWS and Azure, where Polaris keeps its own state, how federation lets it sit in front of catalogs you already run, and the parts of the problem it deliberately refuses to solve.
Iceberg splits a table into data files and metadata files. The metadata tracks which data files belong to the table at a given snapshot, what the schema is, how the table is partitioned, and what has changed over time. That metadata lives in object storage next to the data.
So why do you need a catalog at all? Because something has to answer one specific question atomically: where is the current metadata file for this table?
That pointer is the whole game. A commit in Iceberg is a swap of that pointer from one metadata file to the next. If two writers try to swap it at the same time and both succeed, you have lost a commit. If a reader catches it halfway, it sees a table that never existed. The catalog is what makes that swap atomic, and it is the reason you cannot run a serious Iceberg deployment on a bare filesystem layout.
Polaris does that job. Then it does three more things that a plain pointer store does not: it models permissions over the objects it tracks, it isolates tenants from each other, and it brokers credentials so engines can read storage without holding standing keys.
Picture two Spark jobs writing to the same Iceberg table at 09:00. Both read the current metadata pointer, both write new data files, both produce a new metadata file, and both try to move the pointer. Without a coordinator, the second write silently erases the first. The data files are still sitting in object storage, orphaned and paid for, but no snapshot references them.
The catalog prevents that by making the pointer swap a compare-and-swap. The second writer presents the metadata location it expects to replace, finds that it no longer matches, and fails the commit. Iceberg clients then retry against the new snapshot.
This is the reason a filesystem-only Iceberg setup is a prototype rather than a deployment. Object stores give you eventual consistency on listings and no cross-object transaction. The catalog supplies the guarantee the storage layer will not.
Polaris nests entities in a strict hierarchy, and understanding it makes the rest of the system read clearly.
A catalog is the top-level entity that maps directly to an Apache Iceberg catalog. Every catalog in Polaris is associated with a storage type, which tells Polaris what kind of object store sits underneath and therefore which credential mechanism applies.
A namespace groups tables. Namespaces nest, so analytics.finance.reporting is a valid path. Iceberg clients see these as the multi-level namespaces the REST spec describes.
Tables and views are the leaves. A Polaris table entity is not the data. It is a pointer to Iceberg metadata plus the properties and grants that Polaris layers on top.
One detail from the docs that will save you an incident. Catalog properties are client-visible. Polaris returns them to any authenticated catalog client through the Iceberg REST /config response. Do not put passwords, tokens or access keys in catalog properties. They are configuration, not secrets.
Polaris validates entity names at the REST layer and rejects violations with HTTP 400. A valid name is not empty, is not . or .., contains no ISO control characters, and contains none of these characters:
/ : * ? " < > | # + `It also cannot start or end with whitespace. These rules apply to create, register and rename operations. Entities whose names predate the validation still read and update fine, which matters if you are migrating an older catalog that let odd names through.
Polaris tracks two entity types that are easy to miss and change what the catalog is for.
Generic tables let Polaris register tables that are not Iceberg. If a namespace holds Delta Lake tables or raw file collections alongside Iceberg tables, the catalog can carry a reference to them so the whole namespace is discoverable in one place. You do not get Iceberg's transactional guarantees on those tables, which is the tradeoff, but you do get one inventory instead of two.
Policies are first-class securable objects. They sit in the same permission model as catalogs, namespaces and tables, which means a policy is something you can grant privileges over rather than a setting buried in a config file. Policy names carry stricter naming rules than other entities.
Both of these point at the same direction of travel. The catalog is becoming the place where you describe what data exists and what may be done with it, not just where the Parquet lives.
A realm is Polaris's logical partition. One deployment can host many, and nothing crosses between them.
This is stronger than it sounds. The realm identifier is part of the primary key in the persistence layer, so entities from different realms cannot collide even though they share a database instance. Isolation is enforced by the storage schema rather than by application code remembering to filter.
Principal credentials are associated with a specific realm too. A principal in staging is not a weaker version of a principal in prod. It is a different principal that prod has never heard of.
Polaris resolves the realm from the incoming request. The default resolver reads it from request headers, and everything downstream, including which metastore manager handles the request, follows from that resolution. Realm identifiers also appear in configuration, including connection strings, which is how each realm's data ends up stored separately:
jdbc:postgresql://localhost:5432/{realm}If you are running one Polaris for several teams, realms are the control you want. If you are running one per environment, they save you three deployments.
Here is what actually happens when Spark asks Polaris for a table. This sequence is the clearest way to understand the architecture, because every component appears exactly once.
The engine calls loadTable against the Iceberg REST endpoint. Polaris resolves the entity from its metastore, checks the caller's grants against the securable object, and then does the part that surprises people: it calls out to the cloud provider for temporary credentials, scoped to that table's storage locations and to the operations the caller is actually allowed to perform.
The engine gets back the Iceberg metadata plus a credential bundle. From that point on, the engine reads data files straight from object storage. Polaris is not in the data path and never sees a byte of your table content.
That design choice explains a lot. It is why Polaris can be small. It is why catalog latency shows up in planning rather than scan time. And it is why the credential broker matters so much: it is the only thing standing between a principal and raw storage access.
Before an engine loads anything, it calls the Iceberg REST /config endpoint. Polaris answers with catalog-level configuration and any overrides the client should apply. This is where catalog properties surface, and it is why the docs are blunt about not storing secrets in them: every authenticated client gets this response.
The response also tells the client which storage settings to use. For an S3-compatible store like MinIO or Apache Ozone, that includes the custom endpoint and path-style addressing flag. For native AWS S3, those fields are simply absent and the SDK falls back to defaults.
If a client works against your laptop MinIO and fails against production S3, this handshake is the first place to look. The difference is usually one config property that was present locally and is not present in the cloud.
The Iceberg REST spec has a place for credentials in the LoadTableResult response, but it does not standardise the property names. Polaris uses the names the Iceberg SDK expects for each storage type, and the docs list them exactly. Knowing them turns “authentication is broken” into a five minute fix.
For AWS S3, Polaris calls STS AssumeRole with an inline session policy scoped to the specific table locations and to the read, list and write operations the caller is authorized for. The response carries:
s3.access-key-id credential
s3.secret-access-key credential
s3.session-token credential
s3.session-token-expires-at-ms credential
client.region config
client.refresh-credentials-endpoint config
s3.endpoint config (S3-compatible stores only)
s3.path-style-access config (S3-compatible stores only)Properties marked credential carry sensitive material and come back in the credentials map, which is never logged. Properties marked config come back in the config map. If you are running MinIO or Apache Ozone rather than native S3, the endpoint and path-style properties appear. On native AWS they are absent, which is a useful signal when you are debugging a client that works locally and fails in the cloud.
Azure works differently. Polaris generates a User Delegation SAS token scoped to the container and path prefix the caller can reach. Azure caps User Delegation SAS validity at seven days, which is a platform limit rather than a Polaris choice, and it is worth knowing before you design a long-running job around it.
The pattern underneath both is the same. No engine holds a standing key. Every credential is scoped to a path, limited to permitted operations, and expires. That is a meaningfully different security posture from handing Spark an IAM role and hoping the bucket policy is right.
Polaris stores entities and grant records in a metastore. The supported backends are a relational JDBC store, with PostgreSQL and CockroachDB both documented, and MongoDB for the NoSQL option.
This is the component people under-plan. The metastore is on the critical path for every query's planning phase, and it holds your entire permission graph. Treat it like the operational database it is: back it up, monitor it, and size it for the concurrency your query engines will produce rather than for the number of tables you have.
Polaris ships an admin tool for bootstrapping and maintaining that store, plus a command line interface for day to day catalog operations. There is also a Polaris Console, a benchmarks suite, a synchronizer, and an MCP server for connecting AI agents to catalog metadata.
Most teams asking about Polaris already have a catalog. Usually several. Federation is the feature that makes adoption something other than a migration project.
Polaris can federate to an external Iceberg REST catalog, to a Hive Metastore, or to BigQuery Metastore. The client connects to one Polaris endpoint and sees one catalog. Polaris resolves each namespace to whichever backing catalog owns it.
The practical value is sequencing. You can put Polaris in front of a Hive Metastore that a hundred jobs still depend on, start creating new tables natively in Polaris, and move the old ones when there is time. Nothing has to be rewritten on day one.
Reading an architecture is partly about noticing what is missing. A few absences in Polaris are choices, not gaps.
It does not execute queries. There is no planner, no scan, no join. That work belongs to Spark, Trino, Flink, Dremio or whatever else you point at it. A catalog that also executed queries would be a database, and it would stop being neutral between engines.
It does not move or rewrite data. Compaction, snapshot expiry and orphan cleanup are maintenance operations that engines perform. Polaris tracks the results, it does not perform the work.
It does not define a semantic layer. Polaris knows a table exists and who can read it. It does not know that rev_q3_fnl means quarterly recognised revenue or that finance considers it authoritative. That modelling lives above the catalog.
Each absence keeps the surface small enough that many engines can agree on it. A catalog with opinions about query execution would be a catalog that only one engine implements well, and then you are back to the thing Iceberg was supposed to solve.
Polaris uses role-based access control. Privileges are granted to catalog roles, catalog roles are granted to principal roles, and principal roles are assigned to principals. Privileges are never granted directly to a principal.
Securable objects are catalogs, namespaces, Iceberg tables, views and policies. That list is the full surface over which permissions can be expressed.
For teams with existing policy infrastructure, Polaris supports external Policy Decision Points, including Open Policy Agent. If your authorization rules already live in Rego and you would rather not maintain a second copy, that path exists.
The architectural point is that authorization lives with the catalog. Every engine that connects gets the same answer, because the check happens before a credential is issued. You do not configure Spark permissions and Trino permissions and hope they agree.
Polaris runs as a service. The interesting choices are not about the service itself, they are about what you attach to it.
Relational JDBC with PostgreSQL is the well-trodden path, and CockroachDB is documented if you need a distributed relational store. MongoDB covers the NoSQL option. This decision is hard to reverse and it constrains your availability story, so make it deliberately rather than accepting a default.
Size it for concurrent planning requests, not for table count. A thousand tables queried rarely puts less load on the metastore than fifty tables hit by a BI tool refreshing every minute across two hundred dashboards.
Polaris supports external identity providers, and the documented path uses Keycloak. If you already run OIDC, wire Polaris into it rather than maintaining a second population of principals. The realm model means you can map an identity provider per tenant if your isolation requirements go that far.
Storage connections and the catalog endpoint both carry credentials. The docs cover TLS configuration for storage, and there is a production configuration reference worth reading before you promote a deployment rather than after.
There is also a Helm chart with its own production configuration guidance if you are running on Kubernetes, which covers persistence, services and networking as separate concerns.
No. Polaris stores metadata about your tables and the grant records that control access to them. Your data files stay in object storage, and engines read them directly after Polaris issues a credential. Polaris is never in the data path.
It can be, but it does not have to be immediately. Polaris federates to Hive Metastore, so you can front an existing metastore and migrate namespaces gradually rather than cutting over.
A catalog role holds privileges on objects inside one catalog. A principal role groups people or services. You grant catalog roles to principal roles, and principal roles to principals. Privileges never attach directly to a principal, which is what makes the model auditable.
Long enough for the operation, and no longer. On AWS the lifetime comes from the STS session. On Azure, User Delegation SAS tokens are capped by the platform at seven days. Polaris also returns a refresh endpoint so clients can renew before expiry rather than failing mid-job.
Yes, through realms. Each realm has its own principals, catalogs, roles and grant records, and the realm id is part of the metastore primary key, so the separation is structural rather than a filter applied at query time.
Dremio co-created Polaris and donated it to the ASF. Dremio's Open Catalog is built on it, which means the catalog behind Dremio speaks the same Iceberg REST API as any Polaris deployment, and tables created through Dremio stay readable by Spark, Flink, Trino, StarRocks and Apache Doris without a copy or a connector.
That matters for the argument this whole post makes. A catalog that only your query engine understands is a lock-in surface wearing an open format as a disguise. The Iceberg files may be portable, but if nothing else can find them, portability is theoretical.
Dremio also runs the federated query engine and Reflections on top, so a query can join an Iceberg table in your lake to a table in PostgreSQL without a pipeline in between, and Reflections can accelerate that query by up to 100x without anyone rewriting it. The catalog stays open underneath.
Polaris runs as open source software you deploy yourself, and it also appears as a managed service inside commercial platforms. The architecture described here holds in both cases, because the contract is the Iceberg REST Catalog specification rather than a vendor SDK.
That is the practical test for any catalog claiming openness. Can a different engine, from a different vendor, connect and write? With Polaris the answer is yes by construction, and the list of engines already doing it includes Apache Spark, Apache Flink, Trino, StarRocks, Apache Doris and Dremio.
If you are choosing between running it yourself and consuming it as a service, the decision is about who operates the metastore and the identity integration, not about whether your tables stay portable.
The mistake worth avoiding is treating Polaris as a directory. A directory tells you where things are. Polaris decides who may see them, hands out time-boxed keys to storage, keeps tenants from noticing each other, and makes the one atomic swap that keeps Iceberg tables consistent.
If you take one thing into a deployment plan: the metastore and the credential broker are the two components that will determine whether this works at scale. Everything else is comparatively forgiving.
Stand up 1.7.0 against a local MinIO bucket, load a table from Spark, and watch the credential come back in the response. The architecture makes sense immediately once you have seen that exchange once.
One more framing that helps. Iceberg made the table format open. Polaris makes the control plane open. Those are different problems, and solving only the first gets you portable files that a single vendor still decides who may read.
The projects that matter here are governed in public. You can read the proposals, join the community meetings, and see the roadmap before it ships. For a piece of infrastructure that sits between every engine and every byte you own, that visibility is worth more than any feature comparison.
For a full treatment of the catalog layer, download Apache Polaris: The Definitive Guide by Alex Merced, Andrew Madson, and Tomer Shiran, free from Dremio.
To see these ideas applied in practice, explore Dremio’s Open Catalog, built on Apache Polaris.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]