The Dremio Blog

Partition evolution is one of the features that makes Iceberg a safe long-term choice. It means the partitioning decision you make today is not permanent.

From that point, all engines share a consistent view of your Iceberg tables. New tables created by Spark appear in Dremio immediately. Schema changes committed by Flink are visible to PyIceberg clients without any manual sync. The catalog handles the coordination.

A lakehouse without governance is a liability. Sure, you can query it, but can you trust it? Analysts find tables with no owner, no description, and no clear indication of whether what they're looking at is current. Likewise, compliance teams can't demonstrate data lineage and engineers can't assess the impact of a schema change before […]

Partition evolution is one of those features that seems minor until you need it. Then it's the difference between a two-minute metadata update and a two-day rewrite project. If you're building on Iceberg and haven't thought carefully about your partition strategy yet, the time to do that is before your table reaches 10 TB, not after.

Business intelligence tools are where data stops being infrastructure and starts being useful. Executives review performance in dashboards, product teams track metrics in reports, and finance runs variance analysis against actuals. In each case, the value only materialises if the connection between the tool and the underlying data is fast, reliable, and consistent. Dremio connects […]

The framing that matters here: agentic analytics is not a feature you add to your existing BI stack. It is a different approach to how analytical work gets done, who does it, and at what speed.

The Agentic Lakehouse is not a different architecture from your existing lakehouse. It is four additional structural layers built on top of a foundation you have likely already built: an AI Semantic Layer, Autonomous Performance, active metadata, and agent-specific interfaces.

The release of Apache Polaris 1.5.0 marks a significant step forward in the project's evolution. This release introduces enterprise-grade security integrations, expanded catalog federation, advanced credential vending, and key performance optimizations.

This composability is what makes the Agentic Lakehouse architecture viable long-term. As Iceberg V3 adoption grows and the Polaris REST Catalog becomes the universal standard for catalog interoperability, adding a new engine or a new AI framework to your stack becomes a configuration change, not a migration project.

The single biggest performance advantage of Iceberg over raw data lakes is not a clever algorithm or a faster codec. It is metadata-driven data skipping. By the time a query engine begins scanning actual Parquet files, Iceberg's metadata has already eliminated 90-99% of the files from consideration.

Apache Iceberg V3 is a meaningful advancement over V2, not a version bump for its own sake. Deletion vectors address the fundamental I/O cost of merge-on-read that V2 delete file accumulation creates. The Variant type eliminates one of the most common workarounds in modern data pipelines: storing JSON as strings and parsing at query time.

Apache Iceberg machine learning workflows are at an inflection point. As AI systems become more autonomous, the requirement to audit what data an AI model was trained on shifts from an engineering preference to a compliance requirement. Financial regulators, healthcare compliance frameworks, and emerging AI transparency mandates are moving toward requiring documentation of training data provenance.

The foundation you built today, a connected source, a semantic layer, a documented catalog, and working AI agent interfaces, is the starting point for all of those capabilities. Each addition builds on what you already have rather than requiring a separate system.

The Iceberg ecosystem is consolidating fast. REST Catalog interoperability, growing AI tooling, and the Apache governance model mean that every month you stay on Delta Lake, you are working against the direction of the industry. The migration investment pays off in engine flexibility, catalog portability, and access to a growing set of tools that assume Iceberg as the standard.

This guide walks you through building a complete Dremio semantic layer for an e-commerce analytics use case from scratch. You will connect raw sources, build three tiers of views, add documentation, apply access control, and verify the whole thing works with Dremio's AI Agent.