The Dremio Blog

Dremio also facilitates a gradual and flexible adoption process. Organizations can start small, using only the necessary components, and scale up as their requirements grow. This approach reduces the initial investment and complexity, making it easier for businesses to transition to a data lakehouse architecture at their own pace.

The synergy of Apache Arrow, Apache Iceberg, and Nessie within Dremio simplifies complex data management tasks and democratizes access to data analytics, enabling a more data-driven approach in organizations.

The data lakehouse, as the latest milestone in this evolution, embodies the collective strengths of its predecessors while addressing their limitations. It represents a unified, efficient, and scalable approach to data storage and analysis, promising to unlock new possibilities in data analytics.

Dremio's Lakehouse platform represents a significant step forward in the evolution of data management. By leveraging data virtualization and lakehouse architecture, it offers a viable solution to the limitations of traditional ETL-based approaches. Organizations embracing Dremio can expect an improvement in their data management capabilities and a strategic advantage in the fast-paced world of data-driven decision-making.

Dremio stands as a formidable solution to the pervasive challenge of data silos. Unifying disparate data sources enables organizations to leverage their data assets fully, enhancing decision-making and operational efficiency. As the data landscape evolves, tools like Dremio will be critical in shaping a more integrated and insightful approach to data analytics.

Whether through direct PyArrow library usage or leveraging the dremio-simple-query library for simplified querying and data manipulation, the synergy of these tools opens up new possibilities for data analysis and processing. The ability to convert data streams into different formats ensures compatibility with a wide array of data processing and analytics tools, making this approach highly versatile.

this is a product release announcement regarding new ingestion capabilities for Apache Iceberg. Customers can now use COPY INTO to get data in parquet format into Iceberg tables.

The demand for insightful and high-performance dashboards has never been greater. As organizations accumulate vast amounts of data, the challenge lies in visualizing this data efficiently, especially when dealing with large datasets. In this article, we will delve into the realm of BI dashboards, exploring the hurdles that hinder their performance for sizable datasets. Traditionally, […]

This month, we’re excited to announce automated table cleanup!

The concept of data marts has long been a pivotal strategy for organizations seeking to provide specialized access to critical data for their business units. Traditionally, data marts were seen as satellite databases, each carved out from the central data warehouse and designed to serve specific departments or teams. These data marts played a vital […]

The Databricks platform is widely used for extract, transform, and load (ETL), machine learning, and data science. When using Databricks, it's essential to save your data in a format compatible with the Databricks File System (DBFS). This ensures that either the Databricks Spark or Databricks Photon engines can access it. Delta Lake is designed for […]

We're always looking for ways to better handle and save money on our data. That's why the "data lakehouse" is becoming so popular. It offers a mix of the flexibility of data lakes and the ease of use and performance of data warehouses. The goal? Make data handling easier and cheaper. So, how do we […]

This article has been revised and updated from its original version published in 2022 to reflect the latest developments in all three table formats. The three major open table formats (Apache Iceberg, Delta Lake, and Apache Hudi) each solve the "open lakehouse" problem differently at the architectural level. While the high-level comparison covers features and ecosystem support, […]

Using Airbyte, you can easily ingest data from countless possible data sources into your S3-based data lake, and then directly into Apache Iceberg format. The Apache Iceberg tables are easily readable directly from your data lake using the Dremio data lakehouse platform, which allows for self-service data curation and delivery of that data for ad hoc analytics, BI dashboards, analytics applications, and more.

In the last decade, a lot of innovation in the data space took place to address the need for more storage, more compute, hybrid environments, data management automation, self-service analytics, and lower TCO to operate business at scale.