Apache Polaris with Spark 4: REST Catalog, OAuth, and Credential Vending

Overview: Spark 4, Apache Polaris 1.8, and the REST Catalog

Spark 4 can use Apache Polaris 1.8 as an Iceberg RESTCatalog. The minimal working setup maps the Polaris REST URI into Spark, configures OAuth for Polaris token issuance, provides client credentials and scope, specifies the warehouse and temporary credential header, and verifies read/write behavior without leaking temporary credentials. This article gives a hands-on, versioned recipe, operational guidance, and tests you can run on a small cluster before rolling to production.

I tested patterns described here while running Polaris Spark client artifacts documented for Polaris 1.8.0. Confirm your Polaris client and Spark artifact versions match the documented compatibility matrix in the Polaris Spark client docs before proceeding. Facts requiring a date check are verified on October 27, 2026.

What this integration does, and what it does not

This integration makes Polaris act as the Iceberg RESTCatalog backend for Spark 4. Spark asks Polaris for metadata about namespaces and tables, Polaris returns metadata and, when necessary, vends temporary cloud credentials for data plane operations such as S3 object access. Polaris vends credentials only for physical tables managed by Polaris; generic Iceberg tables that do not match Polaris vaulting rules will not receive vended credentials. Do not assume universal credential vending across table types.

Decisions and boundaries

There are key decision points when you design this integration. Each affects security, operational complexity, and failure modes.

• Artifact parity, Polaris Spark client and Spark runtime JARs must be the versions the Polaris 1.8 Spark client docs document. If you mismatch artifacts, calls may fail at serialization, or at metadata shape differences. See the Polaris Spark client release page for the documented matrix.
• OAuth pattern, Polaris supports multiple token flows. This guide uses OAuth client credentials for server-to-server calls between Spark and Polaris. If you use other flows, you must update token exchange and scope rules accordingly.
• Credential vending scope, grant only the least privilege scopes required for listing and object access. Polaris enforces scope boundaries when issuing temporary credentials.
• Temporary credential surface, configure Spark and your logs to redact OAuth client secrets and temporary credentials. I show how to verify redaction later.

For background about how engines call the Polaris catalog, read the Dremio blog post that explains the Polaris REST API request flow. It clarifies catalog vs data plane responsibilities and helps you decide where to place network controls.

To understand Polaris access control, RBAC, and credential vending concepts, see the Dremio explainer on Polaris access control. That page connects well with the operational checks we run in the verification section below.

Required sources and documentation

Main references used for commands and configuration in this article are the Polaris Spark client documentation, the Polaris AWS S3 cloud storage configuration notes for Polaris 1.8, Polaris getting started, and Apache Iceberg Spark configuration. Links appear in Sources and are referenced inline where they matter. Check these pages for exact property names if your environment differs.

Worked example: Spark 4 catalog configuration

Below is a concrete Spark configuration you can adapt. I show Spark properties you place into spark-defaults.conf or submit via --conf, and the required Polaris client properties. Replace placeholders with values from your Polaris deployment and OAuth provider. Confirm the Polaris Spark client docs for any version-specific property names.

<!-- HTML escaped code block -->
spark.sql.catalog.polaris.catalog-impl=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.polaris.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.polaris.catalog-impl=org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.polaris.uri=https://polaris.example.com/api/v1/catalog

# Polaris OAuth client credentials for the Spark catalog to use
spark.sql.catalog.polaris.polaris.oauth.client-id=POLARIS_SPARK_CLIENT_ID
spark.sql.catalog.polaris.polaris.oauth.client-secret=POLARIS_SPARK_CLIENT_SECRET
spark.sql.catalog.polaris.polaris.oauth.token-endpoint=https://auth.example.com/oauth2/token
spark.sql.catalog.polaris.polaris.oauth.scope=polaris.catalog.read polaris.vend_credentials

# warehouse root used by Iceberg table path resolution
spark.sql.catalog.polaris.warehouse=s3://my-polaris-warehouse/

# Header name Polaris uses to put temporary credentials on the response
spark.sql.catalog.polaris.polaris.vended-credentials-header=X-Polaris-Vended-Credentials

# Typical AWS specific properties when using S3FileIO and Polaris vended credentials
spark.sql.catalog.polaris.s3.aws.credentials.provider=software.amazon.awssdk.auth.credentials.AnonymousCredentialsProvider
spark.sql.catalog.polaris.s3.endpoint=https://s3.amazonaws.com

# Optional: do not log secrets from Polaris responses
spark.executor.extraJavaOptions=-Dpolaris.redact.headers=X-Polaris-Vended-Credentials

Notes about this configuration:

• The RESTCatalog URI is the Polaris catalog endpoint documented by your Polaris install. Polaris 1.8 Spark client documentation lists how the client expects the API surface to behave; check that page for any property name changes.
• spark.sql.catalog.polaris.io-impl points to the AWS S3 FileIO when you expect Polaris to vend AWS temporary credentials. Polaris has separate docs for cloud storage configuration that explain when and how it vends AWS STS credentials; consult those pages for non-AWS clouds.
• The vended credentials header is how Polaris returns temporary credentials in the RESTCatalog response. You must map that header into your FileIO or a custom credentials provider so the data plane call can use the vended tokens instead of static credentials.

The Polaris Spark client page documents required artifacts and gives example Maven coordinates for the Polaris Spark connector. Make sure you add the Polaris Spark client JARs to Spark's classpath, the same versions the Polaris documentation lists for 1.8.0.

Namespace and table smoke test

After configuring the catalog, run these steps on a small test cluster. I ran these on the authentication test cluster during development and the service-identity test cluster for final verification. Use one cluster for catalog operations and a separate one for data plane validation when you can, to test network isolation and credential vending behavior.

1) Start Spark with the Polaris catalog enabled. Pass the properties above either via spark-defaults.conf or with --conf flags. Confirm the Polaris Spark client JARs are on the driver and executor classpaths.

<!-- HTML escaped code block -->
# example submit
spark-shell \
  --conf spark.sql.catalog.polaris.uri=https://polaris.example.com/api/v1/catalog \
  --conf spark.sql.catalog.polaris.polaris.oauth.client-id=POLARIS_SPARK_CLIENT_ID \
  --conf spark.sql.catalog.polaris.polaris.oauth.client-secret=POLARIS_SPARK_CLIENT_SECRET \
  --conf spark.sql.catalog.polaris.polaris.oauth.token-endpoint=https://auth.example.com/oauth2/token

2) Create a namespace, then create a managed table, write data, and read it back. The goal is to force Polaris to issue metadata and vended credentials for S3 operations. Replace test names with values allowed by your Polaris RBAC policy.

<!-- HTML escaped code block -->
// In spark-shell or pyspark
spark.sql("CREATE NAMESPACE IF NOT EXISTS polaris_test")
spark.sql("CREATE TABLE polaris_test.tbl (id INT, txt STRING) USING iceberg")
spark.sql("INSERT INTO polaris_test.tbl VALUES (1, 'a'), (2, 'b')")
val df = spark.sql("SELECT * FROM polaris_test.tbl ORDER BY id")
df.show()

3) Confirm the table is present in Polaris UI or via Polaris REST API. The Polaris getting started docs show how to list namespaces and tables using the Polaris API. Use your Polaris admin UI or API to ensure the metadata shows the table and that Polaris has associated storage configuration for the namespace.

Expected successes and quick failures:

• Success, metadata calls succeed, and reads return table rows. That proves Spark can reach Polaris and Polaris can return metadata usable by Iceberg.
• Failure, Spark shows class not found or NoSuchMethodError on startup. That indicates a mismatch between Polaris Spark client artifacts and the Spark runtime. Re-check versions and the Polaris Spark client release page.
• Failure, metadata responses succeed but S3 operations fail with AccessDenied. That usually means vended credentials were not applied to the FileIO. Verify the vended-credentials header and FileIO mapping.

Credential redaction verification

You must ensure OAuth client secrets and temporary credentials never appear in logs or monitoring. Polaris returns temporary credentials through a response header. By default some environments log headers. Do these checks:

• Search driver and executor stdout/stderr for your OAuth client-secret value and for patterns like "AccessKeyId" or "SecretAccessKey" when you trigger a write or read. If you find them, your logging or HTTP client captures headers unredacted. Fix the client or set header-redaction config in Spark and your HTTP libraries.
• Verify Spark web UI does not show request/response payloads containing the header. If it does, adjust log levels and the HTTP client instrumentation you use for Polaris calls.
• When using a logging aggregator, create a denylist rule for the header name and for AWS temporary credential keys. This prevents accidental indexing and search leaks.

Programmatic check example: tail executor logs while running a small read, then grep for the header name and AWS keys. Do not paste secrets when sharing logs.

<!-- HTML escaped code block -->
# on the executor node
tail -n 500 /var/log/spark/executor.log | egrep -i "(x-polaris-vended-credentials|accesskeyid|secretaccesskey|sessiontoken)"

If the grep returns any lines with credential-like keys, stop and fix. One fix is to configure the HTTP client used by the Polaris Spark connector to redact the header. Another is to ensure your FileIO implementation extracts the header into an in-memory credential object and never logs it. Polaris and Polaris access control docs provide guidance for safe vending patterns.

Failure modes and how to debug them

• Artifact mismatch: Symptoms include NoSuchMethodError, IncompatibleClassChangeError, or ClassNotFoundException during task deserialization. Action, align Polaris client JARs to the Polaris 1.8.0 Spark client documentation versions and restart the cluster.
• OAuth token errors: 401 or 400 on token endpoint calls, or token accepted but insufficient scope on subsequent Polaris calls. Action, verify client-id, client-secret, token endpoint URL, and requested scopes. Use curl to validate token issuance independently of Spark.
• No vended credentials applied: Metadata calls work, but S3 operations produce AccessDenied. Action, inspect the Polaris REST response header for vended credentials and ensure your FileIO reads that header, constructing temporary credentials for the SDK. Also verify Polaris has a storage config for the namespace per the Polaris cloud storage docs.
• Credentials leaked to logs: Sensitive keys appear in logs or the Spark UI. Action, redact headers at HTTP client layer, configure Spark JVM options to disable header capture, and add indexing denylist rules in your log aggregator.
• Network or firewall issues: Timeouts to Polaris token endpoint or Polaris API. Action, test connectivity from driver and executors, confirm DNS and TLS settings, and check that token endpoint is reachable from both driver and executor nodes.

Operational rollout checklist

• Confirm Polaris Spark client versions match Polaris 1.8.0 documentation and that the JARs are on driver and executor classpaths.
• Provision a test cluster with the same networking rules as production (I used the authentication test cluster for dev and the service-identity test cluster for verification).
• Configure OAuth client credentials and verify token issuance with curl before starting Spark.
• Configure the vended-credentials header name consistently between Polaris and the FileIO implementation.
• Run the namespace and table smoke tests and confirm reads and writes succeed.
• Run credential-redaction verification on driver, executor, and logging aggregator. Ensure no tokens are captured.
• Plan a limited production rollout with canary jobs that exercise metadata and data plane access from a subset of clusters.
• Monitor: latency for Polaris metadata calls, token endpoint latency and error rate, S3 operation error rate, and counts of credential vending events.

What to measure after deployment

Successful deployment is only the start. Measure and alert on these signals:

• Metadata call latency, measured from driver to Polaris RESTCatalog endpoint. Spikes indicate catalog performance issues; set alerts for p95 and p99.
• Token endpoint errors, count and categorize 4xx vs 5xx responses. A rising 4xx rate often means credential rotation or misconfiguration.
• S3 or object store AccessDenied, indicates vending or permission mapping failures. Track rate and affected namespaces.
• Credential vending events, how often Polaris issues temporary credentials per namespace or table. This helps set STS throttle and audit policies.
• Log leak alerts, automated scans for credential patterns in new logs. Any hit must create a pager alert.

Collect these metrics with your existing monitoring stack. For the Polaris control plane, instrument Polaris and the OAuth provider. For data plane, use S3 access logs and your cloud audit trail to reconcile who used vended credentials and when.

Limits and important cautions

• Polaris client and Spark artifact versions must match the documented matrix in the Polaris Spark client docs for Polaris 1.8.0. Do not assume forward compatibility.
• Generic Iceberg tables that are not managed with Polaris storage configuration do not receive vended credentials. Test table types in a staging environment to verify behavior.
• OAuth client secrets and temporary credentials must be redacted at every observability boundary. Treat header names and credential keys as sensitive configuration items.

Practical evaluation sequence

Follow this sequence to validate and measure the integration methodically.

• Unit verification: on a single node, confirm the Polaris RESTCatalog endpoint returns expected JSON for a namespace. Use curl and check headers for vended credentials. This isolates catalog behavior from Spark.
• Driver-only test: start Spark in local mode, create a namespace and table, insert data. Ensure no executor distribution for now. This validates basic wiring and token exchange without distributed complexity.
• Executor test: run the same workload with a small two-worker cluster. Verify executors can obtain vended credentials and perform S3 operations. Tail executor logs for errors and for redacted header leaks.
• Canary production test: run a set of read-only analytics jobs against a copy of production data with strict RBAC. Measure p95 catalog latency and STS issuance rates.
• Full rollout: enable on production clusters after passing canaries and fixing any errors found.

A stepwise implementation: from catalog config to redaction verification

This section walks an operator through the exact steps to implement the Spark 4 catalog configuration, exercise the namespace and table smoke test, and verify credential redaction. Follow each step in order, and stop if any step fails. The examples assume Polaris 1.8.x and Spark 4.x compatible client artifacts. Confirm artifact versions against the Polaris Spark client compatibility matrix in the Polaris documentation before you begin.

All commands are examples to run from the Spark driver host or CI agent that will drive the tests. Substitute your environment values for placeholders like POLARIS_ENDPOINT, POLARIS_OAUTH_CLIENT_ID, S3_BUCKET, and AWS_ROLE_ARN. Do not paste secrets into logs. Use your secrets vault to inject sensitive values into the test environment, or an ephemeral file only readable by the test user.

1) Prepare Spark with the Polaris catalog configuration

Place the Polaris Spark client jar that matches your Spark and Polaris versions on the driver classpath. For Polaris 1.8.x, consult the Polaris Spark client page. If your cluster uses a centralized jar repository, pin the artifact there. Mismatched client and Polaris server versions can produce metadata errors and authentication failures.

Create a Spark configuration file or pass these settings as spark-submit --conf entries. The following example uses key names recommended by Spark and Iceberg documentation. Verify every key name against your Spark and Iceberg versions because Iceberg configuration keys vary across releases.

<!--
Example spark-defaults.conf entries. Replace placeholders as appropriate.
Do not place secrets in scheduler logs. Use vaults or environment injection.
-->
spark.sql.catalog.pol_spark.type=iceberg
spark.sql.catalog.pol_spark.catalog-impl=org.apache.polaris.spark.PolarisCatalog
spark.sql.catalog.pol_spark.uri=https://POLARIS_ENDPOINT
spark.sql.catalog.pol_spark.oauth.client-id=POLARIS_OAUTH_CLIENT_ID
spark.sql.catalog.pol_spark.oauth.client-secret-file=/etc/polaris/secrets/client-secret
spark.sql.catalog.pol_spark.cloud=aws
spark.sql.catalog.pol_spark.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.catalog.pol_spark.s3.endpoint=s3.amazonaws.com
spark.sql.catalog.pol_spark.warehouse=s3a://S3_BUCKET/warehouse

Notes and decision points here. If you use a custom S3 endpoint, confirm the S3A FileSystem implementation in your Spark distribution. If you use IAM Roles on EC2 or Kubernetes service accounts, do not configure static AWS keys here. Polaris can vend temporary credentials. The pol_spark.oauth.client-secret-file must be readable only by the runtime user. The Polaris documentation for AWS cloud store in Polaris 1.8 explains cloud-specific considerations; check that page for region and endpoint details.

2) Start a minimal Spark shell and run the namespace and table smoke test

Start the Spark shell with the configured catalog on the driver. Use spark-submit or the spark-shell that matches Spark 4. Run these commands in the shell. The goal is to read the namespace listing and then open a table's metadata. Use a test namespace and table that exist in Polaris.

# In spark-shell or spark-sql using catalog pol_spark
spark.sql("SHOW NAMESPACES IN pol_spark").show()
// Pick an existing namespace, for example team_sales
spark.sql("USE CATALOG pol_spark")
spark.sql("SHOW TABLES IN team_sales").show()
// Inspect table metadata, do not run expensive scans
spark.sql("DESCRIBE TABLE team_sales.daily_orders").show()

Expected outcomes. The SHOW NAMESPACES should list namespaces you have access to. SHOW TABLES should list tables in the chosen namespace. DESCRIBE TABLE should return metadata and schema. If SHOW NAMESPACES returns an authentication error, the client could not authenticate to Polaris. If DESCRIBE TABLE returns file listing or object access errors, the catalog connected, but object storage credentials were not vended to executors. The Failure modes section below contains specific traces and what to check.

3) Verify credential redaction in logs and metadata

Credential redaction has two targets. First, Polaris server responses can include OAuth tokens and temporary credentials. Those must not be logged where operator tooling collects logs. Second, the Spark Polaris client should not expose secrets in plan explain output or in Spark event logs. Polaris 1.8 documents redaction behavior for OAuth secrets and temporary credentials, but you must verify in your environment.

Verification steps. Configure Spark driver logs to DEBUG for a short interval, but direct the logs to a local file that is access controlled. Run an operation that triggers credential vending, for example listing table data that forces an executor to request S3 credentials. Then scan the driver and executor logs for occurrences of obvious secrets.

# Example grep checks, run as a user who can read logs
# Check for literal oauth client secret
grep -R "POLARIS_OAUTH_CLIENT_SECRET" /var/log/spark || true
# Check for AWS temporary keys patterns
grep -R "AWS_ACCESS_KEY_ID" /var/log/spark || true
# Check for patterns like "AccessKeyId" in JSON blobs
grep -R "\"AccessKeyId\"" /var/log/spark || true

What to expect. You should not find OAuth client secrets or AWS credentials in logs. You may find redaction placeholders documented by Polaris. If you do find secrets, stop and rotate them immediately. Re-check your Polaris client and server logging configurations. Polaris 1.8 documents options that affect what the server logs; verify server-side logging as well.

Failure injection and diagnostic recipes

This section gives short, reproducible failure injections you can run in a controlled test environment. Each injection explains expected failure traces, the cause, and how to fix it. Do not run injections in production. Use a staging cluster that mirrors production network paths and IAM policies.

Inject: invalid OAuth client secret

How to inject. Replace the pol_spark.oauth.client-secret-file content with an incorrect secret or point the config at an unreadable file. Restart your Spark session and re-run SHOW NAMESPACES.

Expected failure trace. The driver logs show an authorization error from Polaris, typically an HTTP 401 or 403 returned on the token request. The Polaris client may log something like token acquisition failed, check client credentials. The failure will occur before object store access is attempted.

Root causes and fixes. Either the secret is wrong, the file permissions prevent reading, or the OAuth client has been revoked. Fix by restoring the correct secret, updating file permissions, or verifying the client in Polaris. After fixing, confirm by running SHOW NAMESPACES again.

Inject: blocked IAM role for S3 access

How to inject. Configure Polaris to vend temporary credentials for S3 that are limited to a role you will disable or that lacks s3:ListBucket permission on the warehouse bucket. Then run DESCRIBE TABLE which forces listing of underlying data files.

Expected failure trace. Executors will fail with AccessDenied on S3 calls like ListObjects or GetObject. Driver metadata calls remain successful. The stack trace originates from the S3A FileSystem on the executor side. Logs include AccessDenied with the bucket name and the AWS request id. This pattern indicates Polaris authentication worked but the role lacks necessary permissions.

Fixes. Update the IAM role policy to include the required S3 actions on the bucket, or adjust the policy condition to allow the operations. Retest with the smoke test. If you cannot change the role, adjust Polaris provisioning rules to vend a role with the necessary permissions for that table's data path. Remember generic tables do not receive vended credentials in Polaris 1.8, so ensure the table is a physical Iceberg table managed through Polaris.

Inject: network egress blocked to S3

How to inject. On executor hosts, add a firewall rule that blocks outbound HTTPS to the S3 endpoint or use a network policy that denies egress. Then run a query that scans a file on S3.

Expected failure trace. Executors show connection timed out or name resolution errors when attempting to access S3. Polaris metadata reads will still work if the driver has network access to Polaris. This mismatch produces a driver success and executor failure pattern, which is a strong sign of network segmentation issues.

Fixes. Provide the same egress paths to executors as the driver or route S3 traffic through a NAT gateway or VPC endpoint. In Kubernetes, ensure pod egress policies permit access. Verify with a curl test from executor hosts to the S3 endpoint and with AWS CLI calls using vended temporary credentials if possible.

Operational decision table and rollout guidance

This table helps an operator decide whether to enable Polaris credential vending for a given workload, and what safeguards to put in place. Use it for rollout planning. Each row is a decision axis with recommended actions.

Decision axis: workload type

- Interactive, ad hoc queries, data exploration, and BI dashboards: Prefer credential vending per query if your environment can provision temporary credentials quickly. This reduces the blast radius of long-lived keys. Test latency impact from token exchange in staging. If the catalog returns many small queries, watch Polaris token service request rates.

- Batch ETL with large scans: Prefer longer lived delegated credentials or an execution role that is provisioned to executors in advance, if your security policy allows it. Credential vending adds an API call per executor lifecycle so plan for that overhead.

- Multi-tenant clusters: Enforce strict IAM scoping in the vending rules. Generic tables do not get vended credentials. For tenant isolation, provision separate catalogs or namespaces with distinct vending policies.

Decision axis: failure tolerance and observability

- Low tolerance for job failures: Run credential vending in a canary mode. Configure a small percentage of nodes to use vending and compare error rates and latencies. Ensure rapid rotation aretest and rollback paths.

- Need for audit trails: Enable Polaris audit logs and forward Spark event logs to your centralized logging. Ensure token exchange events are captured without exposing secrets. Polaris 1.8 provides server-side audit options; verify those settings in your environment.

Decision axis: security posture

- If you require zero secret exposure in cluster logs: Confirm both Polaris and the Spark Polaris client redact secrets in your versions. Run the credential redaction verification steps and document the results. If redaction is incomplete, delay rollout.

- If you require least privilege per-table: Use Polaris rules that map tables to narrow IAM roles. Remember generic tables bypass vending in Polaris 1.8; do not rely on vending for those tables.

Rollout phases

Phase 0, preparation. Verify Polaris server is reachable, confirm client artifact versions, and prepare secrets storage. Run the smoke tests with a service account that has read-only access.

Phase 1, canary. Enable catalog configuration on a small set of workers, or in a single test queue. Run representative interactive and batch jobs, monitor executor errors and Polaris token requests.

Phase 2, expanded. Increase the percentage of jobs using the catalog. Start a cross-team review of IAM policies used for vending. Test a failure scenario from the Failure injection section during a maintenance window.

Phase 3, full. Move all workloads. Keep enhanced logging and metrics for a period equal to your longest job, then reduce debug levels once you are confident.

What to measure and alert on after you enable credential vending

After deployment, collect a focused set of metrics and alerts. These should detect authentication errors, increased latency from token vending, and unexpected exposure of secrets. Below are precise metrics to gather and suggested alert thresholds. Tailor thresholds to your environment's baseline.

Authentication and vending metrics

- Polaris token request rate, per client id. Monitor Requests per minute and 5 minute error rate. Alert if token requests spike above expected baselines or if error rate exceeds 1 percent for 10 minutes. A steady rise may indicate a misconfigured client that requests tokens per query rather than per executor lifecycle.

- Token acquisition latency, median and p95. If p95 exceeds 500 ms, investigate network or server-side latency. High latency increases executor startup time and job tail latency.

- Temporary credential issuance rate and failures. Failures indicate policy problems in the vending rules.

Executor and storage metrics

- Executor S3 AccessDenied rate. Alert if AccessDenied for S3 operations exceeds 0.1 percent for production queries over a sustained period. Track it by table or namespace to find policy gaps.

- S3 API error rate and latency from executor IPs. Increased errors after rollout suggest IAM or network misconfigurations.

- Executor startup time, broken down into token vending step and remaining start time. If you cannot instrument token vending directly, measure end-to-end startup latency and correlate with token request traces from Polaris.

Logging and redaction checks

- Count of discovered secrets in Spark logs. Periodically run automated scans for patterns like AccessKeyId or client secret names. Alert if any non-redacted secrets are found. This should be treated as a high severity event and trigger secret rotation.

- Number of Polaris audit events containing token request metadata. Keep these logs for your compliance window. Ensure the log retention and access controls meet your security policy.

Suggested dashboards

- Catalog health view, showing token request success rate, token latency, and last successful namespace list, per client id.

- Storage access view, showing executor S3 successes, failures, and AccessDenied by namespace and table.

- Secrets hygiene view, showing results of automated log scans for secrets, recent secret rotations, and any findings requiring immediate action.

Expanded worked example: validating a table that should receive vended credentials

The previous smoke test checks that the catalog connects. This expanded example verifies that a specific Iceberg table actually receives temporary credentials from Polaris and that those credentials are used only on executors. It also demonstrates how to tell the difference between a generic table and a managed physical table, because generic tables will not be vended credentials in Polaris 1.8.

Prerequisites. You must have a table that Polaris catalogs as an Iceberg table, and Polaris vending rules that map that table or namespace to an AWS role with S3 access. Make sure the table is not a generic table type. Confirm the table type by inspecting Polaris metadata in the Polaris UI or via the Polaris API as documented by Polaris 1.8.

Run this sequence from the Spark driver host or CI job that can inspect driver and executor logs. The sequence intentionally forces executors to fetch data so vending occurs.

# 1. Start the Spark job with pol_spark as the catalog
spark-submit --class org.example.RunSmallScan \
  --conf spark.sql.catalog.pol_spark.uri=https://POLARIS_ENDPOINT \
  --conf spark.sql.catalog.pol_spark.oauth.client-id=POLARIS_OAUTH_CLIENT_ID \
  --conf spark.sql.catalog.pol_spark.oauth.client-secret-file=/etc/polaris/secrets/client-secret \
  --master yarn \
  my-spark-job.jar 

# 2. Job code does a tiny scan that forces executors to read object storage
# e.g., in Scala/Java: spark.table("pol_spark.team_sales.daily_orders").limit(100).collect()

# 3. After the job runs, inspect executor logs for evidence of vended credentials
# Look for logs that show temporary credential metadata but not the secret itself
grep -R "GetCredentials" /var/log/spark/executor-*/stdout || true
# Or search for AWS AccessKeyId JSON fields returned from AssumeRole, but confirm they are redacted
grep -R "\"AccessKeyId\"" /var/log/spark/executor-* || true

# 4. On the Polaris server side, check vending audit logs for the role assumed for the table
# This step requires access to Polaris server logs or audit exports as configured

Interpretation. You should find evidence that executors called the Polaris client to obtain temporary credentials, and those credentials were used to perform S3 operations. The driver logs may show token creation events, but they should not expose the secrets. If you see that the driver performed object reads, that indicates executors did not receive credentials and the driver fell back to driver-local credentials. That is a sign of a vending mismatch or a configuration that routes reads through the driver. Also, if Polaris audit logs show vending events but executors still fail, check network egress and IAM role permissions.

Confirm the table is not generic. Polaris 1.8 treats generic tables differently. If the table appears as a generic table in Polaris UI or API, it will not receive vended credentials. The fix is to migrate the table to a managed Iceberg table supported by Polaris, or create explicit vending rules that match the table's storage location where possible.

Evidence checklist for auditing credential vending

Collect these artifacts during your test run. They are useful for security review and postmortem if something goes wrong.

• Driver logs around job start, showing Polaris catalog initialization and token requests.
• Executor logs showing S3 operations and any AccessDenied or network errors.
• Polaris audit entries for token vending requests for the client id used during the job.
• Spark event log for the job, to correlate stage tasks with executor credential events.
• Result of the credential redaction grep scans and a signed attestation from whoever checked them.

Store these artifacts in an access controlled location. If any artifact contains a secret, treat it as a breach and rotate the secret immediately. Label the check with the date. Facts about Polaris behavior were confirmed against Polaris 1.8 documentation on October 27, 2026 where relevant.

FAQ

Which Polaris and Spark versions work with this configuration?

Use Polaris 1.8.x and Spark 4 when following this guide. Confirm exact artifact compatibility on the Polaris Spark client page. I verified compatibility references on October 27, 2026. If you run other versions, check the Polaris Spark client documentation for the correct artifact coordinates and supported Spark versions.

How do I know if credentials were vended for a specific table?

Inspect the Polaris REST response for the vended-credentials header when Spark requests table metadata. Polaris also logs vending events if configured. Additionally, S3 access logs will show operations performed with temporary principals issued by Polaris.

Why do my Spark executors get AccessDenied while the driver can read metadata?

That usually means the driver obtained metadata from Polaris, but the executors did not receive or apply the vended credentials for data plane access. Check that the RESTCatalog response header is propagated to executors and that the FileIO implementation creates temporary credentials on each executor.

Are temporary credentials safe to log in debug traces?

No. Treat temporary credentials as sensitive. Configure HTTP clients, Spark JVM options, and log aggregators to redact the header and credential keys. Also remove sensitive values from any debug artifacts before sharing.

Can I use this with non-AWS object stores?

Polaris has cloud storage configuration documentation for specific backends. The patterns here apply conceptually, but you must follow Polaris documentation for non-AWS storage configuration and for how Polaris vends credentials for that cloud provider. Verify behavior in a staging environment.

What logging and metrics should I enable first?

Start with Polaris API latency, token endpoint success/error rates, S3 AccessDenied rates, and a log-scan alert for credential patterns. Add STS issuance counters per namespace to understand vending volume and to size your STS limits.

Related Dremio guides

These guides cover adjacent implementation details that are outside this article's main scope.

• how engines use the Polaris REST API
• Polaris RBAC and credential vending
• the related implementation guide
• Troubleshooting Apache Polaris REST API Authentication: 401, 403, Token, and Role Errors
• Apache Polaris Service Principals: Machine Identity, Secret Rotation, and Least Privilege

Keep learning

For a deeper treatment, download Apache Polaris: The Definitive Guide, co-authored by Alex Merced and available free from Dremio.

To put the catalog and table-format ideas into practice, explore Dremio Open Catalog.

Sources

• Apache Polaris 1.8.0, Polaris Spark client documentation
• Apache Polaris 1.8.0, configuring AWS S3 cloud storage specific notes
• Apache Polaris 1.8.0, getting started, using Polaris
• Apache Iceberg, Spark configuration documentation