The Dremio Blog

The convergence of AI agents, open table formats, and semantic tooling is making this architecture decision more consequential than it was a few years ago. AI agents that query through ungoverned raw tables or that cannot discover what data exists are not reliable.

The most expensive mistake in data lake querying is the accidental full table scan: a query that reads every file because the user did not correctly reference the partition columns. In Hive, this happens constantly. In Iceberg, it is structurally impossible because users never reference partition columns at all.

 Semantic layer governance AI is the architectural pattern that closes this gap by enforcing data access controls structurally, at the layer every query must pass through, rather than procedurally, in workflows that agents simply skip.

In data analytics, it's the query engine that gets all the attention. It's where the SQL runs and where the performance story is told. But the storage layer underneath is just as important; it's the "lake" part of the "lakehouse" after all. Choose the wrong storage infrastructure and you're facing I/O bottlenecks no query engine […]

The reason so many agentic analytics projects stall at proof-of-concept is not the AI model. It is the absence of the infrastructure that would make the AI trustworthy on real data. A semantic layer is that infrastructure.

The Model Context Protocol (MCP) changes this equation. An MCP server data lakehouse setup gives any compliant AI client a single, governed, structured gateway to your data. You configure it once. Every agent that follows the spec connects automatically.

Solving the Apache Iceberg small files problem requires addressing it at multiple layers. Detection comes first: use table_files() to establish a baseline and set thresholds that trigger action. Prevention comes next: configure write.target-file-size-bytes at the source and increase checkpoint intervals for streaming jobs.

Partition evolution is one of the features that makes Iceberg a safe long-term choice. It means the partitioning decision you make today is not permanent.

From that point, all engines share a consistent view of your Iceberg tables. New tables created by Spark appear in Dremio immediately. Schema changes committed by Flink are visible to PyIceberg clients without any manual sync. The catalog handles the coordination.

A lakehouse without governance is a liability. Sure, you can query it, but can you trust it? Analysts find tables with no owner, no description, and no clear indication of whether what they're looking at is current. Likewise, compliance teams can't demonstrate data lineage and engineers can't assess the impact of a schema change before […]

Partition evolution is one of those features that seems minor until you need it. Then it's the difference between a two-minute metadata update and a two-day rewrite project. If you're building on Iceberg and haven't thought carefully about your partition strategy yet, the time to do that is before your table reaches 10 TB, not after.

Business intelligence tools are where data stops being infrastructure and starts being useful. Executives review performance in dashboards, product teams track metrics in reports, and finance runs variance analysis against actuals. In each case, the value only materialises if the connection between the tool and the underlying data is fast, reliable, and consistent. Dremio connects […]

The framing that matters here: agentic analytics is not a feature you add to your existing BI stack. It is a different approach to how analytical work gets done, who does it, and at what speed.

The Agentic Lakehouse is not a different architecture from your existing lakehouse. It is four additional structural layers built on top of a foundation you have likely already built: an AI Semantic Layer, Autonomous Performance, active metadata, and agent-specific interfaces.

The release of Apache Polaris 1.5.0 marks a significant step forward in the project's evolution. This release introduces enterprise-grade security integrations, expanded catalog federation, advanced credential vending, and key performance optimizations.