Arctic Changelog – July 2023

Dremio Arctic is a lakehouse management service that makes data engineering easier, faster, and cheaper than ever. With Arctic, data teams can support development, test, production, and data science workloads on a single, no-copy lakehouse environment:

• Create branches for development and testing workloads, instead of physical copies of your environment.
• Run ETL pipelines and make changes on isolated dev/UAT branches before instantly promoting changes to production, instead of physically replicating changes between environments.
• Immediately rollback changes to recover from mistakes, instead of spending hours trying to identify and undo errors.
• Create temporary branches for data scientists to run experiments, instead of creating physical copies for them.

This month, we’re excited to announce new features that help you enrich, accelerate, and secure your data in Arctic!

Wikis and Labels

You can now add helpful context to folders and datasets for teammates using wikis and labels.

Use wikis to provide detailed descriptions and context for your datasets, just like a README for your data. Wikis support a GitHub-flavored markdown for formatting. For example, you can add a description of what the dataset contains, the sources it was created from, and provide example queries.

Use labels to easily categorize datasets for users. For example, you can add a “Gold” label to a dataset to indicate it’s of sufficient quality for reporting purposes, or a “Marketing” label to indicate the data originates from the Marketing department.

Reflections

One of key Dremio technologies that enables companies to run interactive BI directly on the data lake is Reflections. Reflections are materializations that are aggregated, sorted, and partitioned in different ways, and transparently accelerate queries — like indexes on steroids. Reflections help end users analyze large datasets quickly and easily, and make life easier for data engineers by eliminating redundant data pipelines and physical data copies.

You can now accelerate queries on any Arctic dataset with reflections.

Arctic also makes it easy to experiment with reflections without disturbing production workloads. For example, you can create branches to experiment with different reflection types and patterns, and see how each reflection affects your workload. Once you’re happy, you can roll out these changes in your production branch to help end users accelerate their workloads!

Check out this Gnarly Data Waves episode to learn more about Reflections!

Role-Based Access Control

You can now secure access to your Arctic catalog using Role-Based Access Control (RBAC) privileges.

With RBAC privileges, you can control which roles and users can access (including read, write, and create branches) and manage your Arctic catalog.

What's Next?

We're excited to build the future of data engineering with Arctic, and we’re encouraged by the feedback we’ve gotten from customers so far. Stay tuned for more updates and new features!

Interested in Learning More?

• Get started for free on Dremio Cloud
• Learn more about Dremio Arctic
• Visit the Project Nessie website

Already Using Iceberg and Want to Test Out Arctic?

• Use the Iceberg catalog migration CLI tool to register or migrate tables from an existing catalog to Arctic (without copying data!) — read the blog to learn more