The Dremio Blog

Central to the functionality of Apache Iceberg tables is their catalog mechanism, which plays a crucial role in the evolution of how these tables are used and their features are developed. In this article, we will take a deep dive into the topic of Apache Iceberg catalogs.

Dremio offers a powerful solution for unifying data across Snowflake, Azure, AWS, and Google based data marketplaces while mitigating egress costs and simplifying data management. By leveraging Dremio's reflections and advanced lakehouse capabilities, you can enhance your analytics without the hassle of complex data movements. We invite you to get hands-on and explore the full potential of Dremio through the tutorials listed below. Discover how Dremio can transform your data operations and take your analytics to the next level.

Dremio's `COPY INTO` command, and the soon-to-be-released Auto Ingest feature provide robust solutions for importing these files into Apache Iceberg tables. By leveraging Dremio, ingesting and maintaining data in Apache Iceberg becomes manageable and efficient, paving the way for performant and flexible analytics directly from your data lake. In this article, we’ll do a hand-on exercise you can do in the safety of your local environment to see these techniques at work.

Dremio enables directly serving BI dashboards from Apache Druid or leveraging Apache Iceberg tables in your data lake. This post will explore how Dremio's data lakehouse platform simplifies your data delivery for business intelligence by doing a prototype version that can run on your laptop.

This exercise hopefully illustrates that setting up a data pipeline from Kafka to Iceberg and then analyzing that data with Dremio is feasible, straightforward, and highly effective. It showcases how these tools can work in concert to streamline data workflows, reduce the complexity of data systems, and deliver actionable insights directly into the hands of users through reports and dashboards.

The most straightforward area to address in terms of reducing costs is your BI Dashboards. Whenever someone interacts with a BI dashboard that uses Snowflake as the data source, queries are sent to Snowflake, increasing your expenditure. Imagine if you could significantly cut the costs of serving dashboards from your Snowflake data by drastically reducing the amount of Snowflake compute resources needed.

Moving data from source systems like MySQL to a dashboard traditionally involves a multi-step process: transferring data to a data lake, moving it into a data warehouse, and then building BI extracts and cubes for acceleration. This process can be tedious and costly. However, this entire workflow is simplified with Dremio, the Data Lakehouse Platform. Dremio enables you to directly serve BI dashboards from MySQL or leverage Apache Iceberg tables in your data lake.

Moving data from source systems like Elasticsearch to a dashboard traditionally involves a multi-step process: transferring data to a data lake, moving it into a data warehouse, and then building BI extracts and cubes for acceleration. This process can be tedious and costly. However, this entire workflow is simplified with Dremio, the Data Lakehouse Platform. Dremio enables direct serving of BI dashboards from Elasticsearch or leveraging Apache Iceberg tables in your data lake.

By leveraging a lakehouse architecture, organizations can achieve significant savings on storage and compute costs, streamline transformations with virtual modeling, and enhance data accessibility for analysts and scientists.

The Dremio Unified Analytics Platform for Analytics and AI hosted our 6th Annual Subsurface Live event this week - the only event dedicated to Lakehouse learning. We are excited to share a few of the exciting announcements, developments, and new capabilities! Dremio for Every Environment Dremio has always been the most flexible lakehouse deployment, with […]

ical use cases. While you can deploy Dremio as self-managed software in a Kubernetes environment, you can get some nice bonuses when working with a Dremio Cloud Managed environment

PuppyGraph's sponsorship underlines our dedication to empowering individuals and organizations with knowledge and tools to navigate and excel in the evolving data landscape. Through its innovative platform and active participation in the community, PuppyGraph continues to lead the way in advancing graph analytics and data lakehouse technologies, making the complexities of big data more accessible and manageable.

By sponsoring the Subsurface Conference, we aim to connect with the community, share insights, and explore the future of data lakehouses together. Join us at the conference to witness the evolution of data management and take your first step towards an optimized data future with Apache Iceberg and Upsolver.

Tim Hurski, Prashanth Badari, Sonal Chavan, Dexin Zhu and Dmitry Chirkov

Dremio's version 25 is not just an update; it's a transformative upgrade that redefines the standards for SQL query performance in lakehouse environments. By intelligently optimizing query processing and introducing user-friendly features for data management, Dremio empowers organizations to harness the full potential of their data, driving insightful business decisions and achieving faster time-to-value. With these advancements, Dremio continues to solidify its position as a leader in the field of data analytics, offering solutions that are not only powerful but also practical and cost-effective.