Using Apache Polaris with PyIceberg: Iceberg from Python, No Cluster

PyIceberg is a pure Python implementation of the Apache Iceberg table format. It talks to Apache Polaris over the Iceberg REST Catalog API, authenticates with an OAuth client credential, and reads and writes tables through Arrow with no JVM and no cluster. It is the right tool for metadata inspection, scripted maintenance, and filtered reads that fit in memory. It is the wrong tool for a distributed join.

Most Iceberg tooling assumes you have a Spark cluster. That assumption is fine when you are processing a terabyte and absurd when you want to know how many snapshots a table has.

PyIceberg removes the assumption. It is a Python library that speaks the Iceberg format and the REST catalog protocol directly, so a script on your laptop can inspect a production table, expire its snapshots, evolve its schema, or pull a filtered slice into a DataFrame, without provisioning anything.

This covers connecting it to Polaris, the configuration that actually matters, reading and writing, the metadata work it is uniquely good at, and where its limits are.

What This Covers

• What PyIceberg is, and is not
• Installing with the right extras
• Connecting to Polaris
• Configuration in a file instead of in code
• Reading, and where the filtering happens
• Appending, overwriting and upserting
• Metadata inspection, the best reason to use it
• Scripted maintenance
• Schema and partition evolution
• Where the limits are
• FAQ

What PyIceberg Is, and Is Not

PyIceberg implements the Iceberg specification in Python. It reads the metadata tree, plans scans, applies filters, reads Parquet through Arrow, and writes data files and commits them through a catalog. There is no query engine underneath, which is exactly the point.

What that buys you is a five second feedback loop instead of a cluster start. What it costs you is distributed compute. A scan that returns four hundred million rows will do so into the memory of one process, and that process will die.

The boundary is not blurry, and treating it as blurry is the source of most bad experiences with the library. Filtered reads, metadata work and modest writes: excellent. Anything that would need a shuffle: use an engine.

Installing with the Right Extras

PyIceberg's dependencies are optional by design, so a minimal install cannot do much:

# the common combination for talking to a REST catalog and reading data
pip install "pyiceberg[pyarrow,s3fs]"

# add whichever consumers you want
pip install "pyiceberg[pyarrow,s3fs,duckdb,pandas]"

# on ADLS or GCS instead
pip install "pyiceberg[pyarrow,adlfs]"
pip install "pyiceberg[pyarrow,gcsfs]"

If a scan raises an error about a missing FileIO implementation, this is why. The catalog call succeeded and there is nothing installed that can read the storage scheme the metadata points at.

Connecting to Polaris

Polaris implements the Iceberg REST Catalog specification, so this is the standard REST catalog configuration with Polaris-specific values:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "polaris",
    **{
        "type": "rest",
        "uri": "https://polaris.internal:8181/api/catalog",
        "warehouse": "analytics_catalog",
        "credential": f"{CLIENT_ID}:{CLIENT_SECRET}",
        "scope": "PRINCIPAL_ROLE:ALL",
        "header.X-Iceberg-Access-Delegation": "vended-credentials",
        "token-refresh-enabled": "true",
    },
)

print(catalog.list_namespaces())

The four values people get wrong

The URI ends in /api/catalog. Polaris serves its management API under /api/management and the Iceberg REST API under /api/catalog. PyIceberg speaks only the second. A 404 on list_namespaces is almost always this.

warehouse is a catalog name. Not a path, not a bucket. It selects which catalog on the Polaris server you are addressing. Storage locations are configured on the catalog, which is what lets the client stay ignorant of them.

credential is client-id:client-secret. PyIceberg exchanges it for a bearer token at the token endpoint, which defaults to the catalog URI plus v1/oauth/tokens. If your Polaris deployment authenticates through an external identity provider, set oauth2-server-uri to point at that instead.

scope is PRINCIPAL_ROLE:ALL for most cases. It requests all principal roles the principal holds. Narrow it when you deliberately want a session that carries less than the principal's full entitlement.

Vended credentials are the point

The header.X-Iceberg-Access-Delegation setting tells Polaris that the client accepts temporary storage credentials. Polaris then returns credentials scoped to the table's location, and PyIceberg uses them to read the data files.

Leave it out and PyIceberg needs its own storage credentials, which means every analyst running a script needs bucket access, which means the permissions you configured in Polaris describe metadata and nothing else. The delegation header is what makes a catalog grant the actual control point.

Configuration in a File Instead of in Code

Hardcoding a client secret in a script is how secrets end up in git. PyIceberg reads .pyiceberg.yaml from the current directory, your home directory, or PYICEBERG_HOME:

catalog:
  polaris:
    type: rest
    uri: https://polaris.internal:8181/api/catalog
    warehouse: analytics_catalog
    credential: ${POLARIS_CLIENT_ID}:${POLARIS_CLIENT_SECRET}
    scope: PRINCIPAL_ROLE:ALL
    header.X-Iceberg-Access-Delegation: vended-credentials
    token-refresh-enabled: true

Then the code is one line:

catalog = load_catalog("polaris")

Every setting also works as an environment variable with a PYICEBERG_CATALOG__ prefix, which is the cleaner option in a container where you would rather not ship a config file at all.

Reading, and Where the Filtering Happens

This is the section that decides whether PyIceberg feels fast or feels broken.

table = catalog.load_table("events.clicks")

arrow_table = table.scan(
    row_filter="event_date >= '2026-09-01' AND country = 'US'",
    selected_fields=("event_id", "user_id", "url"),
    limit=100_000,
).to_arrow()

df = arrow_table.to_pandas()

The filter passed to scan() is pushed down through every layer. Partition predicates eliminate whole manifests before any data file is opened. Column bounds stored in the manifests eliminate data files. Parquet row group statistics eliminate row groups. What reaches your process is what survived all of it.

Compare that to the version people write first:

# downloads the entire table, then throws most of it away
df = table.scan().to_pandas()
df = df[(df.event_date >= '2026-09-01') & (df.country == 'US')]

Same result, and on a large table the second version transfers hundreds of gigabytes to discard almost all of it. Every filter you can express in the scan belongs in the scan.

You can also inspect the plan before running it, which is a good habit on an unfamiliar table:

scan = table.scan(row_filter="event_date >= '2026-09-01'")
files = list(scan.plan_files())
print(f"{len(files)} data files, "
      f"{sum(t.file.file_size_in_bytes for t in files) / 1e9:.1f} GB")

If that prints forty thousand files, stop and reconsider before calling to_arrow(). The plan is cheap and the read is not.

Handing off to something that queries

PyIceberg's job ends at producing Arrow. What consumes that Arrow is your choice:

# DuckDB, for SQL over the result without leaving the process
con = table.scan(row_filter="country = 'US'").to_duckdb(table_name="clicks")
con.execute("SELECT url, count(*) FROM clicks GROUP BY 1 ORDER BY 2 DESC").fetchall()

# Polars, for a lazy DataFrame
lf = table.scan(row_filter="country = 'US'").to_polars()

# plain pandas
df = table.scan(row_filter="country = 'US'").to_pandas()

The DuckDB path is the one worth knowing. A filtered Iceberg scan into DuckDB gives you real SQL, including joins and window functions, over a slice that fits in memory, with no cluster involved. For a large fraction of ad hoc analysis that is the entire requirement.

Appending, Overwriting and Upserting

PyIceberg writes, and the write path goes through the catalog like any other client, so Polaris enforces the same grants it enforces for Spark.

import pyarrow as pa

df = pa.Table.from_pylist(
    [{"city": "Amsterdam", "inhabitants": 921402},
     {"city": "Paris", "inhabitants": 2103000}],
    schema=table.schema().as_arrow(),
)

table.append(df)

Partial overwrite deletes what matches a filter and appends the new rows in one commit:

table.overwrite(df, overwrite_filter="city = 'Paris'")

Upsert merges an Arrow table by identifier field, updating rows that match and inserting rows that do not:

table.upsert(df)   # requires identifier_field_ids on the schema

Two things to keep in mind. The Arrow schema has to line up with the Iceberg schema, and the nullability of fields is part of that, which is the most common cause of a confusing type error on the first write. And these writes are single process, so this is a mechanism for landing thousands or millions of rows, not billions.

Metadata Inspection, the Best Reason to Use It

This is where PyIceberg earns its place even in shops with plenty of Spark. Every metadata table is available as Arrow, from a script, in under a second.

table.inspect.snapshots().to_pandas()
table.inspect.files().to_pandas()
table.inspect.manifests().to_pandas()
table.inspect.partitions().to_pandas()
table.inspect.refs().to_pandas()
table.inspect.history().to_pandas()

A few queries worth keeping around. Small file distribution, which tells you whether compaction is overdue:

files = table.inspect.files().to_pandas()
print(files.file_size_in_bytes.describe())
print(f"under 32 MB: {(files.file_size_in_bytes < 32*1024*1024).mean():.0%}")

Snapshot growth, which tells you whether expiry is keeping up:

snaps = table.inspect.snapshots().to_pandas()
print(len(snaps), "snapshots,", snaps.committed_at.min(), "to", snaps.committed_at.max())

And refs, which is where you find the tag somebody created during an incident and never removed, quietly pinning old snapshots against expiry.

Running these across every table in a catalog is a twenty line script, and it produces a genuine health report for a lakehouse. Doing the same thing in Spark means a cluster and a job.

Scripted Maintenance

PyIceberg can expire snapshots, which means routine maintenance no longer requires an engine at all:

from datetime import datetime, timedelta

table.maintenance.expire_snapshots().older_than(
    datetime.now() - timedelta(days=7)
).commit()

# or target specific snapshots
with table.maintenance.expire_snapshots() as expire:
    expire.by_id(3821553639163471111)

The semantics are the Iceberg semantics, not a Python approximation. Files still referenced by a retained snapshot are not deleted, and expiry is a commit that goes through Polaris like any other. What expiry does and refuses to do is covered in the snapshot expiration article.

One thing to be clear about: PyIceberg does not do compaction or orphan file cleanup at the time of writing. Those still belong to a distributed engine, because both involve reading and rewriting a lot of data or listing a lot of storage. A realistic split is scheduled expiry from Python and periodic compaction from Spark.

Schema and Partition Evolution

Both are transactional and both read naturally:

with table.update_schema() as update:
    update.add_column("referrer", StringType())
    update.rename_column("url", "page_url")

with table.update_spec() as update:
    update.add_field("event_time", DayTransform(), "event_day")

Iceberg's schema evolution is metadata only, so adding a column does not rewrite a single data file. Partition evolution changes the layout for future writes while existing data keeps its old layout, and queries read across both correctly.

Doing this from a Python script rather than a Spark job matters more than it sounds. Schema changes become something a migration script can own, reviewed like any other code, instead of something someone runs by hand in a notebook.

Permissions the Script Needs in Polaris

A PyIceberg script authenticates as a Polaris principal, and that principal reaches privileges through a chain: principal, principal role, catalog role, privilege. The practical question is which privileges an analysis script actually needs.

For read-only work, the answer is less than people usually grant:

# a read-only role scoped to one namespace
polaris principals create analyst_scripts
polaris principal-roles create analyst
polaris principal-roles grant analyst --principal analyst_scripts

polaris catalog-roles create --catalog analytics_catalog events_reader
polaris privileges namespace grant \
  --catalog analytics_catalog --catalog-role events_reader \
  --namespace events TABLE_READ_DATA
polaris catalog-roles grant events_reader \
  --catalog analytics_catalog --principal-role analyst

Two distinctions matter here and both show up clearly in a Python workflow.

Listing is not reading. Polaris separates the privilege to enumerate tables and read their metadata from the privilege to read their data. A discovery script that catalogues what exists needs the first and not the second, which means you can let it walk the whole catalog without handing it a key to anything.

Reading metadata is not reading files. Every table.inspect.* call in this article works with metadata access alone, because the manifests hold the file paths, sizes and column bounds. A table health report across an entire catalog can run with no data access at all, which is a useful thing to know when someone asks why a monitoring job needs credentials.

The full model, including how credential vending derives its scope from these grants, is in the Polaris access control article.

A Health Report Across Every Table

Here is the script that justifies keeping PyIceberg installed. It walks a catalog and reports the things that predict trouble:

from pyiceberg.catalog import load_catalog

catalog = load_catalog("polaris")
SMALL = 32 * 1024 * 1024

for ns in catalog.list_namespaces():
    for ident in catalog.list_tables(ns):
        t = catalog.load_table(ident)
        files = t.inspect.files().to_pandas()
        snaps = t.inspect.snapshots().to_pandas()
        refs  = t.inspect.refs().to_pandas()
        small = (files.file_size_in_bytes < SMALL).mean() if len(files) else 0
        print(f"{'.'.join(ident):40} "
              f"{len(snaps):>6} snapshots  "
              f"{len(files):>7} files  "
              f"{small:>4.0%} small  "
              f"{len(refs) - 1:>2} extra refs")

Three columns, three different problems. A high snapshot count means expiry is not keeping up. A high proportion of small files means compaction is overdue and query planning is paying for it. Extra refs beyond main means something is pinning snapshots, which is why expiry may be deleting less than you expect.

Run it weekly, keep the output, and you have a trend line for a lakehouse that most teams never build because they assume it requires a monitoring product. It requires twenty lines of Python and metadata read access.

Server-Side Scan Planning

A newer part of the REST specification is worth knowing about, because it changes where the planning work happens.

Normally the client fetches manifests and works out which data files match the filter. When a catalog advertises server-side planning, it does that work and returns the file list. PyIceberg uses it automatically when the catalog reports scan-planning-mode=server, either for the whole catalog or per table in the load response, and it polls asynchronous plans until they finish.

For a Python client the appeal is obvious. Planning a scan over a table with fifty thousand manifests means fetching a lot of Avro before you read a single row, and a server that already has the metadata warm does it faster. Whether it is available depends on your catalog and its version, so check rather than assume.

Where the Limits Are

It is single process

Everything happens in the memory of one Python process. A filtered read is bounded by whatever you selected, and an unfiltered read of a large table is bounded by nothing. Check plan_files() when you are unsure.

No compaction, no orphan cleanup

Both need distributed work. Keep an engine around for them, or use a platform that runs them for you.

Write throughput is modest

Fine for a service landing events, wrong for a bulk load. If the write is measured in hundreds of gigabytes, it belongs somewhere else.

Feature parity trails the Java implementation

The Java library is the reference and PyIceberg follows it. The gap is smaller every release and is mostly in newer or less common features. Check the current documentation before assuming a specific capability exists.

Where Dremio Fits

Dremio co-created Apache Polaris and donated it to the Apache Software Foundation, and Dremio's Open Catalog is built on Polaris. Because everything here goes through the Iceberg REST Catalog API, a table a PyIceberg script creates is immediately visible to Dremio, Spark, Flink or Trino, with no registration step.

The pattern that tends to emerge is a sensible one. Python handles the metadata work, the scripted maintenance, and the analysis that fits in memory. A query engine handles the distributed reads and the compaction. The catalog is what keeps them describing the same tables.

Try It on a Real Table

Connect to your Polaris catalog and run three lines against your largest table: count the snapshots, describe the file sizes, and list the refs.

You will learn more about the health of that table in ten seconds than a dashboard will tell you in a week, and you will not have started a cluster to do it. That is the whole argument for keeping PyIceberg in the toolkit even when you have engines available.

Frequently Asked Questions

Does PyIceberg need Spark or a JVM?

No. It is a pure Python implementation of the Iceberg format and reads data through Arrow. That is the reason to use it. The trade is that there is no distributed compute underneath, so everything runs in one process.

How do I connect PyIceberg to Apache Polaris?

Use the REST catalog type with the Polaris catalog URI ending in /api/catalog, the catalog name as warehouse, and an OAuth client credential in the form client-id:client-secret. Add header.X-Iceberg-Access-Delegation: vended-credentials so Polaris supplies temporary storage credentials.

Why is my scan so slow?

Almost always because the filter is applied after the read rather than inside it. Pass row_filter and selected_fields to scan() so the predicate is pushed down through manifests and Parquet statistics. Filtering a pandas DataFrame afterwards means everything was transferred first.

Can PyIceberg compact files or remove orphans?

Not at present. It can expire snapshots, which covers routine metadata maintenance, but compaction and orphan cleanup need distributed work and belong to an engine such as Spark or to a managed platform.

Is it safe to write to a production table from PyIceberg?

Yes, within its scale. Writes go through the catalog and use the same atomic commit protocol as any other Iceberg client, so concurrency is handled correctly. The constraint is volume, not safety.

Where should credentials live?

In .pyiceberg.yaml with environment variable substitution, or in environment variables using the PYICEBERG_CATALOG__ prefix. Not in the script.

Do I need a separate Polaris client library?

No. Polaris implements the Iceberg REST Catalog specification, so PyIceberg's built-in REST catalog support is the client. The only Polaris-specific parts are the URI path, the scope value, and the catalog name in warehouse.

Keep learning

For a full treatment of the catalog layer, download Apache Polaris: The Definitive Guide by Alex Merced, Andrew Madson, and Tomer Shiran, free from Dremio.

To see these ideas applied in practice, explore Dremio’s Open Catalog, built on Apache Polaris.