Announcing Dremio 4.8

Today we are excited to announce the release of Dremio 4.8!

This month’s release delivers multiple features such as external query, a new authorization service API, AWS Edition enhancements and more. This blog post highlights the following updates:

• External query
• Default reflections
• Runtime filtering GA
• Documented JMX metrics and provided sample exporters
• Ability to customize projects in Dremio AWS Edition
• Support for Dremio AWS Edition deployments without public IP addresses

External Query

Every database has its own unique set of features and functions that users want to leverage. In this month’s release Dremio includes a very powerful feature that enables end users to have much more control over how Dremio works with external relational sources.

External query enables users to specify the exact SQL operation to execute on an underlying source. This has many advantages, including the ability to leverage database-specific functions of the underlying datasets that are not currently available in Dremio. It also allows users to structure pushdowns, joins and other types of operations to ensure maximum optimization of their queries.

This feature is structured as a TABLE function which goes against the data source using the following syntax:

<figure class="highlight">
<pre><code class="language-shell" data-lang="shell"></code></pre>
<table class="rouge-table">
<tbody>
<tr>
<td class="gutter gl">
<pre class="lineno">1</pre>
</td>
<td class="code">
<pre>TABLE<span class="o">(</span>.EXTERNAL_QUERY<span class="o">(</span>''<span class="o">))</span> </pre>
</td>
</tr>
</tbody>
</table>
<pre><code class="language-shell" data-lang="shell"></code></pre>
<p><code class="language-shell" data-lang="shell"></code><code class="language-shell" data-lang="shell"></code><code class="language-shell" data-lang="shell"></code><code class="language-shell" data-lang="shell"></code></p>
</figure>

Users can leverage external query as part of a virtual dataset (VDS) definition, or as a SQL statement against any relational data sources and ARP connectors. The “select_statement” within the “EXTERNAL_QUERY” function is not processed by Dremio which gives users the freedom to write any statement that the underlying database supports.

We are very excited about external query because it opens up many different capabilities for the end user. To learn more about external query, check out our documentation.

Default Reflections

Providing the fastest query execution time is a top priority for Dremio. In this release Dremio introduces “default reflections,” an impactful performance enhancement that makes planning faster in scenarios where VDS definitions are very complex. This feature is enabled by default so users don’t have to configure anything to start using it.

Runtime Filtering

To improve performance in traditional data warehouse schemas, Dremio 4.8 supports runtime filtering. Runtime filtering enables Dremio to dynamically apply filters from a smaller joined table to a larger table in order to enhance filtering on larger tables and significantly increase performance. This key capability offers up to 100x improved performance when working with traditional snowflake or star schemas. Best of all, the feature is automatically enabled and requires no tuning or administration. Instead, Dremio simply applies filters from a joined table as they are generated.

Documented JMX Metrics and Provided Sample Exporters

Many organizations use standardized tools to monitor applications throughout the whole organization by extracting and centralizing telemetry information for analysis. The Dremio platform offers rich telemetry information on all aspects of system operation and status. In this release we document how to configure Dremio to export telemetry information to popular services, including Prometheus, CloudWatch and others.

Dremio AWS Edition Enhancements

Dremio 4.8 also adds enhancements to Dremio AWS Edition. Starting in this release, users will be able to launch projects in environments where non-public IPs are required. Additionally, Dremio added the ability to customize projects through custom configuration files the same way users would do it in an on-prem environment.

Learn More

For a complete list of additional new features, enhancements, changes and fixes, please review the release notes. As always, we look forward to your feedback. Please post any questions or comments on our community site.