Announcing Dremio 2.0 – Starflake Reflections, REST APIs, and more!
Today we shipped Dremio 2.0!
This is a major release that includes many new features, performance improvements, and stability enhancements - see the highlights and more details below. Our products team will host a live discussion of these new features - you can register here. We’ll send you a recording later in case you can’t attend live.
- Starflake Reflections. When creating reflections on datasets with joins, Dremio will automatically accelerate queries that involve a subset of the joins. For example, if a user creates a reflection on a dataset that joins a fact table with three dimension tables, Dremio can accelerate queries that include any subset of these joins (e.g. fact table joined with just one of the dimension tables), without having to define multiple reflections.
- New Reflection Management Engine. The new reflection maintenance engine provides improved scalability, debuggability and resilience. The engine automatically optimizes prioritizing, ordering, and queueing reflection refreshes, as well as sophisticated error recovery. These improvements reduce management overhead, speed reflection updates, and significantly reduce resource utilization for maintenance tasks.
- Expanded REST APIs. You can now perform most operations via REST APIs, including issuing queries and retrieving results as JSON; browsing and managing the data catalog; managing reflections; manually refreshing a reflection; and accessing the status and results for a specific job.
- External Reflections. Dremio’s reflection framework now supports the ability to leverage summary tables or other digests managed in external systems (eg, Parquet files created by Spark or Hive). Datasets in any of the data sources that Dremio supports may be leveraged to accelerate queries by Dremio’s cost-based optimizer. External reflections are defined using SQL commands.
- Crowd-sourced dataset acceleration votes. Admins now have the ability to see all datasets that have user acceleration votes to understand acceleration demand and dataset popularity. This is available on the UI and via the Vote API. This feature is limited to Dremio Enterprise Edition.
- Source configuration change warnings. Dremio now warns users when they make configuration changes to sources that will cause existing reflection, format, and sharing settings to be cleared.
- Improved performance for managing reflections on Amazon S3. Upload performance and memory usage characteristics when creating reflections on Amazon S3 has been greatly improved.
- Optimized IN clause performance. SQL IN clause performance has been greatly improved in Dremio’s SQL execution engine and query planning for push-downs.
Learn more in the release notes here: https://docs.dremio.com/release-notes/20-release-notes.html
You can download here: https://www.dremio.com/download
Be sure to follow the upgrade instructions for your deployment method: https://docs.dremio.com/advanced-administration/upgrade.html
If you have questions, please post on: https://community.dremio.com