Announcing the Dremio October 2021 Release

Today, we’re excited to announce our Dremio October 2021 (19.0) release!

This month’s release expands our support for analyzing Apache Iceberg tables, broadens our support for analytical SQL functions, and establishes Dremio as the first query engine outside of AWS to integrate with AWS Lake Formation.

AWS Lake Formation Integration

Companies can configure Dremio to verify data access privileges with various authorization clients. For example, Dremio supports Ranger-based authorization for Hive data sources, where Dremio checks Ranger permissions at runtime to allow or disallow access to Hive data.

Within the AWS ecosystem, AWS Lake Formation provides access controls for datasets in AWS Glue, and is used by customers as a central location to define security policies that are shared across multiple tools. With this release, we’re excited to announce that customers can now leverage access permissions defined in Lake Formation to provide secure access to Glue datasets.

Once configured, Dremio verifies with Lake Formation at runtime if a user can access a Glue dataset.

We’re particularly excited about our new Lake Formation integration, as it establishes Dremio as the first query engine outside of AWS to integrate with Lake Formation. This is just one of many areas where we partner closely with Amazon to deliver integration between our services, collaborate on new features, and ultimately enable AWS customers to build next-generation data architectures.

For more details and instructions on how to enforce Lake Formation permissions on new and existing Glue sources, check out the Lake Formation page in the Dremio docs.

Broader Coverage for Analytical SQL Functions

Aside from developing first-class integration with leading ecosystem tools, we’re always working to empower companies to run analytics on their data as fast and as easy as possible. With this release, customers can now run a broader range of analytical SQL queries, such as filtered aggregates and PIVOT/UNPIVOT.

Extending Dremio’s SQL support makes analytics easier for users. For example, consider the following aggregation query, which calculates average selling prices for sales regions, where sales regions are defined on column values that meet a certain criteria:

SELECT
    SUM(CASE WHEN region IN 'NE', 'SE' THEN selling_price ELSE 0.0 END) / SUM(CASE WHEN region IN 'NE', 'SE' THEN 1.0 ELSE 0.0 END) AS EastASP,
    SUM(CASE WHEN region IN 'NW', 'SW' THEN selling_price ELSE 0.0 END) / SUM(CASE WHEN region IN 'NW', 'SW' THEN 1.0 ELSE 0.0 END) AS WestASP,
    SUM(CASE WHEN product_type = 'installed' THEN selling_price ELSE 0.0 END) / SUM(CASE WHEN product_type = 'installed' THEN 1.0 ELSE 0.0 END) AS InstalledASP
FROM source.customer_transactions WHERE ...

Customers can now greatly simplify the above query to:

SELECT
    AVG(selling_price) FILTER (WHERE region IN ('NE', 'SE')) AS EastASP,
    AVG(selling_price) FILTER (WHERE region IN ('NW', 'SW')) AS WestASP,
    AVG(selling_price) FILTER (WHERE product_type = 'installed') AS InstalledASP
FROM source.customer_transactions WHERE ...

For more information on Dremio’s SQL coverage, check out the SQL Functions page in the Dremio docs.

Apache Iceberg Support

Apache Iceberg is an open source table format for analytical datasets. Iceberg enables multiple applications to work together on the same data in a transactionally consistent manner, and defines additional information on the state of datasets as they evolve over time.

With this release, customers can now leverage Dremio’s native Iceberg support to analyze Iceberg tables which leverage a Hive Metastore. Dremio’s high-performance reader automatically identifies which datasets are saved in the Iceberg format, and utilizes the table information from Iceberg manifest files. We recommend using Hive Metastore as a metadata catalog when working with Iceberg tables in production environments.

This feature is generally available. If you’d like to learn more about how Dremio supports Apache Iceberg, visit the Apache Iceberg page in the Dremio docs.

For a technical deep-dive on Iceberg, check out our write-up Apache Iceberg: An Architectural Look Under the Covers.

Learn More

We’re excited about the features and improvements we’ve made this month! For a complete list of additional new features, enhancements, changes, and fixes, check out the Release Notes. And, as always, we look forward to your feedback — please post any questions or comments on the Dremio Community!

P.S. — we just announced limited availability of Dremio Cloud, our fully-managed SQL lakehouse platform! We’d love to have you be a part of this journey. Join our limited availability program today for free!