The Dremio Blog

A data lakehouse is an architecture: object storage, an open table format such as Apache Iceberg, a catalog, and multiple engines reading the same tables. Medallion architecture is a layering convention: bronze for raw data, silver for cleaned data, gold for business-facing aggregates. They are not alternatives. You run medallion inside a lakehouse, and the question worth arguing about is not which to pick but how many of those layers need to be physical copies.

PyIceberg is a pure Python implementation of the Apache Iceberg table format. It talks to Apache Polaris over the Iceberg REST Catalog API, authenticates with an OAuth client credential, and reads and writes tables through Arrow with no JVM and no cluster. It is the right tool for metadata inspection, scripted maintenance, and filtered reads that fit in memory. It is the wrong tool for a distributed join.

Apache Flink writes to Apache Iceberg tables through a catalog, and Apache Polaris is an Iceberg REST Catalog implementation, so Flink connects to it with 'catalog-type'='rest' and an OAuth client credential. Polaris authenticates the job, enforces what it may write, and hands back short-lived storage credentials. Every successful Flink checkpoint becomes one Iceberg snapshot, which is the single most important thing to understand before putting this in production.

An orphan file is a file sitting in an Apache Iceberg table's storage location that no snapshot references, past or present. It is usually the debris of a write that failed partway through. Snapshot expiration cannot remove it, because expiration works from the metadata outwards and the metadata has never heard of it. Finding orphans requires listing storage and comparing that listing against everything the metadata knows about.

Snapshot expiration is the Apache Iceberg maintenance operation that removes old table versions from metadata and deletes the data files only those versions referenced. It is the reason a delete statement eventually reduces your storage bill. It does not delete anything a retained snapshot still needs, and it costs you the ability to time travel to whatever it removed.

Iceberg 1.11.0, released in May 2026, contains the full implementation of the v3 spec(ification), including features like deletion vectors, the VARIANT type, and row lineage. But what comes next? V4, obviously, as four comes after 3. However, it's still early days on the v4 spec, so we are a ways off from the next version […]

Apache Parquet is the recommended file format used in every modern data platform, and for good reason. But what are those reasons? And would it really matter if you stuck with CSV? The short answer is "YES". The slightly longer answer is "Yes, because columns". The full answer is below, so keep on reading to […]

For most of its history, the semantic layer was considered a solved problem. You built it once, business users queried it wherever it lived, and (hopefully) everyone would agreed on what "revenue" meant. However, much like the information in your data dictionary, the popularity of the semantic layer went stale and businesses turned to new, […]

Data pipelines used to require a lot of infrastructure to keep running: separate compute for transformation, staging layers between systems, and a growing stack of tools to manage it all. Dremio changes the equation. With native ingestion, flexible transformation, and AI-assisted pipeline development, teams can build and operate end-to-end ELT workflows directly in the lakehouse, […]

Most conversations about AI and data platforms start with MCP. That's understandable: the Model Context Protocol has become the standard way to give AI agents a window into a data system, and Dremio's MCP server does this well. But MCP solves the specific problem of giving agents a supervised, conversational interface to your data. What […]

Learn the benefits of agentic analytics and how enterprise teams use natural language queries, governed data and AI agents to improve decisions.

The insurance industry is undergoing a structural shift. What was once a slow moving, data heavy sector is now being reshaped by real time intelligence, automation, and advanced analytics powered by artificial intelligence. Agentic AI is no longer a futuristic concept or a “nice to have” innovation, it is rapidly becoming the competitive baseline that […]

 Semantic layer governance AI is the architectural pattern that closes this gap by enforcing data access controls structurally, at the layer every query must pass through, rather than procedurally, in workflows that agents simply skip.

In data analytics, it's the query engine that gets all the attention. It's where the SQL runs and where the performance story is told. But the storage layer underneath is just as important; it's the "lake" part of the "lakehouse" after all. Choose the wrong storage infrastructure and you're facing I/O bottlenecks no query engine […]

A lakehouse without governance is a liability. Sure, you can query it, but can you trust it? Analysts find tables with no owner, no description, and no clear indication of whether what they're looking at is current. Likewise, compliance teams can't demonstrate data lineage and engineers can't assess the impact of a schema change before […]