How Dremio delivers fast Queries on Object Storage: Apache Arrow, Reflections, and the Columnar Cloud Cache

Object storage is cheap and effectively unlimited, but it was built for durability rather than for query latency. Dremio closes that gap with three mechanisms working together: Apache Arrow keeps data in a columnar in-memory format so no serialization is needed between stages, Reflections maintain query-optimized materializations the engine substitutes automatically, and the Columnar Cloud Cache (C3) keeps frequently read data on local NVMe so repeat scans never go back to object storage.

Dremio is a pioneering data lakehouse platform, renowned for its high-speed query engine. What sets Dremio apart is its ability to execute queries directly on data lake storage, eliminating the need to transfer data to other systems. This capability is powered by technologies like Apache Arrow, reflections, and the Columnar Cloud Cache (C3).

Dremio's architecture is designed for scalability. Whether scaling horizontally by adding more instances or vertically with different-sized engines, Dremio offers unparalleled scalability. This flexibility ensures that businesses of all sizes can harness the power of their data without the limitations of traditional data management systems. The result is a platform that accelerates data queries and enhances data analytics operations' overall efficiency and performance.

Why Object Storage Is Slow to Query

Understanding the three mechanisms is easier once the problem is clear. Object storage behaves differently from the local disks that analytical engines were originally designed against, in three ways that matter for queries.

• Every read is a network request. Fetching a byte range from S3 or ADLS costs a round trip measured in milliseconds, against microseconds for a local NVMe read. A scan that touches thousands of files pays that cost thousands of times.
• There is no cheap random access. Object stores serve whole objects or byte ranges, not pages. Reading a narrow slice of a wide table still means issuing many small requests, and request count is itself a billed dimension.
• The data arrives in a format the engine cannot use directly. Bytes read from storage have to be decoded into whatever in-memory representation the engine works with, and if that representation differs at each stage the same data gets converted repeatedly.

Each of the three mechanisms below attacks one of these. Arrow removes the conversion cost, Reflections remove the need to scan the raw data at all for common query shapes, and C3 removes the network round trip for data that has already been read once.

Apache Arrow: Revolutionizing In-Memory Data Processing

At the heart of Dremio's high-speed data processing capabilities lies Apache Arrow, a standard in-memory columnar format. Apache Arrow excels in fast in-memory data processing, enabling quick loading data from formats like Apache Parquet. This rapid data processing is crucial for businesses that require real-time analytics and insights.

One of the most significant advantages of Apache Arrow is the Apache Arrow Flight protocol. This protocol revolutionizes the transport of columnar data between systems. Unlike traditional data transfer methods that require serialization and deserialization between columnar and row-based formatting, Arrow Flight enables end-to-end transport of columnar Arrow data. This approach dramatically increases performance over conventional JDBC/ODBC connections, making data transfers faster and more efficient.

Reflections: Optimizing Data Queries with Intelligent Representations

Reflections in Dremio change how data querying works. They allow the creation of optimized representations of datasets or views in any Dremio-connected source. These representations are materialized as Iceberg tables on your data lake and are highly customizable. Users can choose which columns to materialize, how to partition and sort the data, and what measures or dimensions to store for aggregation results.

The power of reflections lies in Dremio's intelligent query engine. When a dataset or any view created from it is queried, Dremio can intelligently determine if any available reflections can be used to speed up the query. This means that the entire query or portions can be executed more efficiently. Furthermore, with the introduction of incremental reflection refresh and the reflection recommender, Dremio enhances the freshness of reflections and suggests optimizations based on your query patterns. This improves query performance and ensures that the data remains up to date and relevant.

The Columnar Cloud Cache (C3): Enhancing Performance with In-Memory Caching

The Columnar Cloud Cache (C3) is a key feature in Dremio's architecture, designed to boost query performance dramatically. C3 is an in-memory cache located on the Dremio cluster nodes, which plays a critical role in managing frequently accessed data. Caching this data on the nodes' NVMe storage, C3 effectively reduces the need to repeatedly fetch data from object storage.

This caching mechanism offers two primary benefits. First, it significantly cuts down the network request costs, as less data needs to be transferred over the network. Second, it enhances query performance by providing faster access to frequently used data. The in-memory nature of C3 means that data retrieval is much quicker compared to fetching it from remote object storage, leading to a noticeable improvement in query response times.

Summary: Realizing Cost-Effective and Efficient Data Management

Integrating technologies like Apache Arrow, reflections, and the Columnar Cloud Cache (C3) in Dremio's platform brings a new era in query performance on the data lake. The benefits of these technologies extend beyond just improved query performance; they contribute to a more cost-effective and efficient data management strategy.

Faster query speeds mean that compute resources are utilized more efficiently, leading to a reduction in compute costs as less time and power are needed to process data. Moreover, the reduced need for data transfer and the efficient use of network resources contribute to lower network costs.

Additionally, Dremio's ability to query data directly on data lake object storage opens up new possibilities for data utilization. It reduces the need for expensive data warehousing solutions, allowing organizations to do more with their data without incurring additional costs.

In conclusion, Dremio's innovative approach to data querying and management elevates performance and aligns with the cost and efficiency needs of modern businesses. By using these technologies, organizations can get more out of their data, making data-driven decisions faster, more effectively, and more economically sustainable.

Create a Prototype Data Lakehouse Laptop with this Tutorial

Keep learning

For a deeper treatment of the topics in this post, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.

To see these ideas applied in practice, explore how Dremio approaches the open data lakehouse.

Common Questions

Does querying object storage directly mean giving up warehouse performance?

Not with these three mechanisms in place. The performance gap between a warehouse and a lakehouse comes mostly from the storage round trip and from format conversion, and both are addressable without copying data into a proprietary system.

How is a Reflection different from a materialized view?

A materialized view usually has to be queried by name. A Reflection is substituted by the query planner automatically, so queries written against the original dataset benefit without being rewritten.

Does C3 replace Reflections?

No, they solve different problems. C3 makes reading the same data again cheaper. Reflections make the work itself smaller by pre-computing a query-optimized representation. They compound.