The Dremio Blog

Dremio’s unique ability to support cross-environment queries and accelerate them with reflections enables businesses to leverage a true lakehouse architecture, where data can be stored in the most suitable environment — whether on-premises or in the cloud — and accessed seamlessly through Dremio.

Organizations are constantly pressured to unlock data insights quickly and efficiently while controlling costs. However, as businesses amass ever-increasing volumes of data across multiple environments—both on-premises and in the cloud—they encounter significant challenges with traditional data architectures. A Hybrid Iceberg Lakehouse, such as the one offered by Dremio, delivers substantial Total Cost of Ownership (TCO) […]

Organizations are striving to build architectures that manage massive volumes of data and maximize the insights drawn from it. Traditional data architectures, however, often fail to handle the scale and complexity required. The modern answer lies in the data lakehouse, a hybrid approach combining the best aspects of data lakes and data warehouses. This blog […]

Flexibility and simplicity in managing metadata catalogs and storage solutions are key to efficient data platform management. Nessie’s REST Catalog Implementation brings this flexibility by centralizing table management across multiple environments in the cloud and on-prem, while PyIceberg provides an accessible Python implementation for interacting with Iceberg tables. In this blog, we’ll walk through setting […]

Artificial Intelligence (AI) has become essential for modern enterprises, driving innovation across industries by transforming data into actionable insights. However, AI's success depends heavily on having consistent, high-quality data readily available for experimentation and model development. It is estimated that data scientists spend 80+% of their time on data acquisition and preparation, compared to model […]

Join the Dremio/dbt community by joining the dbt slack community and joining the #db-dremio channel to meet other dremio-dbt users and seek support. Version control is a key aspect of modern data management, ensuring the smooth and reliable evolution of both your data and the code that generates insights from it. While code versioning has […]

Learn more about Apache Polaris by downloading a free early release copy of Apache Polaris: The Definitive Guide along with learning about Dremio's Enterprise Catalog powered by Apache Polaris. The Apache Polaris (incubating) lakehouse catalog is the next step in the world of open lakehouses built on top of open community-run standards. While many other […]

Join the Dremio/dbt community by joining the dbt slack community and joining the #db-dremio channel to meet other dremio-dbt users and seek support. The ability to transform raw data into actionable insights is critical. As organizations scale, they need efficient ways to standardize, organize, and govern data transformations. This is where dbt (data build tool) […]

Operational analytic capabilities are foundational to delivering the personalized experiences that customers expect. While first and third-party market data are often the natural starting point, organizations are increasingly discovering that second-party data—the information exchanged securely and confidentially through partnerships and other strategic vendor relationships — are the differentiator that elevates the customer experience to new […]

Join the Dremio/dbt community by joining the dbt slack community and joining the #db-dremio channel to meet other dremio-dbt users and seek support. Maintaining a well-structured and version-controlled semantic layer is crucial for ensuring consistent and reliable data models. With Dremio’s robust semantic layer, organizations can achieve unified, self-service access to data, making analytics more […]

Efficiency and reliability are paramount when dealing with the orchestration of data pipelines. Whether you're managing simple tasks or complex workflows across multiple systems, orchestration tools can make or break your data strategy. This blog will explore how orchestration plays a crucial role in automating and managing data processes, using tools like CRON and Apache […]

Introduction to the Hybrid Data Lakehouse Organizations are increasingly challenged to manage, store, and analyze vast data. While effective in the past, more than traditional data architectures is needed to meet the demands of modern data workloads, which require flexibility, scalability, and performance. This is where the concept of the hybrid data lakehouse comes into […]

In this tutorial, you'll learn how to use Dremio's Reflections to accelerate query performance. We'll walk through the process of setting up a Dremio environment using Docker, connecting to sample datasets, running a complex query, and then using Reflections to significantly improve the query’s performance. Further Reading on Reflections Step 1: Spin Up a Dremio […]

Designing an optimal partitioning strategy for your data is often one of the most challenging aspects of building a scalable data platform. In traditional systems, data engineers frequently partition data by multiple columns, such as date and another frequently queried field. However, this can result in too many small files or partitions, which ultimately leads […]

Change Data Capture (CDC) is a design pattern used in databases and data processing to track and capture data changes—such as insertions, updates, and deletions—in real-time. Instead of periodically extracting entire datasets, CDC focuses on capturing only the data that has changed since the last update. This approach is crucial in modern data architectures, where […]