The Dremio Blog

The rapid growth of the use of AI throughout the modern data stack has transformed how organizations extract insights from their data. With our latest release, we're excited to announce the general availability of AI Functions — a capability that brings the power of Large Language Models (LLMs) directly into SQL execution, making Dremio’s Agentic […]

Discover how Dremio’s Next Generation Cloud and enterprise MCP Server simplify data exploration with AI-driven queries, governance, and natural-language SQL.

Dremio is always striving to abstract away the physical concerns of data, whether the storage location, partitioning schema, or file size optimisation. Thanks to features such as Data Federation, Iceberg Clustering, and Autonomous Performance functionalities, Dremio users get highly-performant access to their data no matter where it lives.  One of the components that delivers this […]

AI data management is the practice of preparing, organizing, governing, and serving enterprise data so it can be used effectively by AI models and agents. It includes collecting data from multiple systems, maintaining high data quality, enforcing governance, and delivering fast, consistent access to that data for training and inference.

AI-ready data is structured, governed, and accessible in a way that supports machine learning, large language models (LLMs), and real-time intelligent agents. Unlike traditional analytics data, it’s not just clean, it’s optimized for rapid, automated decision-making at scale. AI-ready data supports diverse formats, is accessible without ETL, and maintains the context required to train and operate intelligent systems.

Apache Polaris 1.2.0 continues to make the case for a fully open, production-grade Iceberg catalog. These changes reflect real-world needs: better control, stronger security, broader compatibility, and early hooks for observability. As Iceberg adoption grows, Polaris is becoming the default choice for teams who want to avoid vendor lock-in while building modern lakehouse infrastructure. Whether you’re using Dremio Catalog or deploying Polaris yourself, this release brings features that support scale, safety, and flexibility.

Formats like Lance, Nimble, and Vortex show that innovation is accelerating. Each one introduces new ideas about how to structure, index, and access data. But none of them exist in isolation. They need to integrate with engines, catalogs, and governance layers to be truly useful.

The New Economics of Data Cloud data warehouses like Amazon Redshift were built for a world that no longer exists. In that earlier era, organizations focused primarily on structured business intelligence, static dashboards, and predictable workloads. Data was tightly controlled, compute resources were fixed, and dynamic scalability for rapidly changing workloads was not a concern. […]

Running Polaris locally with MinIO gives you a hands-on view of how an open catalog governs Iceberg tables. But when it’s time to move beyond experimentation, deploying and managing a full Polaris environment can take more effort. That’s where Dremio’s integrated catalog comes in. Dremio’s catalog is built directly on Apache Polaris. It delivers all the same open governance, authentication, and interoperability while removing the setup work. You get a production-ready Polaris deployment with a complete user interface, automated security management, and fine-grained access controls, all without maintaining separate services.

Caching dramatically reduces latency and computational costs by storing frequently accessed data closer to where it's needed. Instead of repeated expensive operations - such as fetching from object storage, planning complex queries, or executing SQL - the data you need is provided in fast, local memory. To deliver on this, Dremio implements different layers of […]

The Shift from Warehouses to the Agentic Lakehouse Amazon Redshift has long been a dependable data warehouse for analytics, but the analytics landscape has evolved. Organizations are no longer just running dashboards—they’re powering agentic AI systems that reason, act, and make autonomous decisions based on live business data. These workloads demand real-time responses, high concurrency, […]

This only scratches the surface of Dremio's content. Explore Dremio University and the Dremio Blog to find much more great Dremio Content. Also get involved in the Dremio and OSS community at developer.dremio.com.

Performance management for Apache Iceberg tables isn’t just about cleaning up small files, it’s about ensuring your data layout evolves in step with your ingestion patterns and query workloads. Dremio’s OPTIMIZE command provides the precision engineers need: merging, splitting, and reclustering data into efficient layouts while keeping metadata lean. With its flexible parameters, you can tailor compaction jobs to strike the right balance between optimization depth, runtime, and cost. At the same time, Dremio’s auto-optimization features mean you don’t always have to run these jobs manually. By letting Dremio continuously monitor and optimize Iceberg tables in the background, your most critical datasets stay query-ready without the overhead of constant maintenance.

The real secret to minimizing Iceberg table maintenance isn’t running more optimization jobs, it’s writing smarter data from the very beginning. By combining batch and streaming ingestion best practices, designing thoughtful partitioning and clustering strategies, tuning table properties, and monitoring file health, you can dramatically reduce the frequency and cost of downstream operations like OPTIMIZE.

Apache Iceberg’s snapshot model is a game-changer for time travel, auditing, and recovery, but it comes with a responsibility: old data must be managed carefully. Without proactive cleanup, tables can accumulate unnecessary files, driving up storage costs, slowing queries, and even creating compliance risks. Dremio’s VACUUM TABLE command provides the control data engineers and architects need to: Expire outdated snapshots, keeping only the versions that align with retention policies. Permanently remove deleted data to meet GDPR and CCPA requirements. Clean up orphan files to ensure storage remains lean and predictable.