Iceberg 1.11.0, released in May 2026, contains the full implementation of the v3 spec(ification), including features like deletion vectors, the VARIANT type, and row lineage. But what comes next? V4, obviously, as four comes after 3. However, it’s still early days on the v4 spec, so we are a ways off from the next version […]
Apache Parquet is the recommended file format used in every modern data platform, and for good reason. But what are those reasons? And would it really matter if you stuck with CSV? The short answer is “YES”. The slightly longer answer is “Yes, because columns”. The full answer is below, so keep on reading to […]
For most of its history, the semantic layer was considered a solved problem. You built it once, business users queried it wherever it lived, and (hopefully) everyone would agreed on what “revenue” meant. However, much like the information in your data dictionary, the popularity of the semantic layer went stale and businesses turned to new, […]
Most conversations about AI and data platforms start with MCP. That’s understandable: the Model Context Protocol has become the standard way to give AI agents a window into a data system, and Dremio’s MCP server does this well. But MCP solves the specific problem of giving agents a supervised, conversational interface to your data. What […]
In data analytics, it’s the query engine that gets all the attention. It’s where the SQL runs and where the performance story is told. But the storage layer underneath is just as important; it’s the “lake” part of the “lakehouse” after all. Choose the wrong storage infrastructure and you’re facing I/O bottlenecks no query engine […]
A lakehouse without governance is a liability. Sure, you can query it, but can you trust it? Analysts find tables with no owner, no description, and no clear indication of whether what they’re looking at is current. Likewise, compliance teams can’t demonstrate data lineage and engineers can’t assess the impact of a schema change before […]
Business intelligence tools are where data stops being infrastructure and starts being useful. Executives review performance in dashboards, product teams track metrics in reports, and finance runs variance analysis against actuals. In each case, the value only materialises if the connection between the tool and the underlying data is fast, reliable, and consistent. Dremio connects […]
A lakehouse is only as useful as the data inside it. Query performance, governance, and semantic layers all depend on one assumption: that the underlying data is accurate, complete, and behaving as expected. When it isn’t, dashboards return wrong answers, AI agents reason from bad inputs, and engineering teams spend days diagnosing problems that should […]
Most analytics workloads are fine with millisecond timestamps. A daily sales report doesn’t care whether a transaction landed at 14:32:07.001 or 14:32:07.001423. But for the teams that do care about sub-millisecond precision, the lakehouse has historically been a weak link. Timestamps either got truncated on ingestion, stored as strings to preserve precision, or shoehorned into […]
Adding a column to a large production table used to require a plan. You’d write the migration script, schedule a maintenance window, kick off a backfill job that rewrote every data file to include the new column, and then wait. For a table with billions of rows on a busy lake, that wait could stretch […]
Your organisation has probably already had the “AI agents” conversation. Maybe it was at a board meeting, maybe it surfaced during quarterly planning, or maybe a team came to you with a proposal and a timeline. Either way, the conversation almost certainly centred on the AI: which model, which vendor, which use case. Very few […]
For all the many improvements data lakehouses bring to analytics, there’s one uncomfortable trade-off: deleting rows is expensive. In a system built around immutable Parquet files, a delete is actually a rewrite. You read the file, filter out the rows you don’t want, and write a new file. At scale those I/O costs mount up […]
Most data teams think about lineage at the table or column level. Which pipeline wrote to this table? Which upstream source feeds this column? Those are useful questions, but they stop short of what actually matters in an audit or incident investigation: which specific rows were affected, by which operation, and when. Apache Iceberg v3 […]
Dremio offers three distinct integration points to the data in your lakehouse. This gives users the freedom to pick the interface, models, and tools that are right for them. Whether you’re a business user, a seasoned data analyst, or a developer, we have an integration that will suit how you like to work. The built-in […]
Working with JSON in an Iceberg lakehouse has always been a compromise: you either store JSON as VARCHAR strings and accept the performance hit every time a query needs to extract a field, or you flatten the JSON into a wide table of nullable columns and watch your schema bloat. Both work fine but have […]
Personality typing is a time-honoured tradition in tech. You’ve taken the Myers-Briggs, argued about whether you’re really an INTJ, and been sorted into a primary-coloured group that allegedly encapsulates your work persona. But the test nobody was brave enough to publish is the one that actually matters to your daily work: which Dremio feature matches […]
Ask a typical data engineering team what percentage of their company’s data they can actually query. The honest answer will be low, probably around 20%. The rest sits in object storage as PDFs, email threads, customer feedback forms, call transcripts, maintenance logs, and scanned documents. It’s all there. It’s all stored. And for the purposes […]
Most companies process terabytes of JSON daily, yet querying it often requires brittle pre-processing pipelines and rigid data contracts. This has analysts and data engineers wasting hours defining explicit schemas just to run a simple aggregation. Dremio eliminates this friction by allowing you to query JSON directly in the lakehouse with complete schema-on-read flexibility. Without […]
For decades, the traditional data experience has been defined by friction, with business teams frequently required to wait. Waiting for SQL experts to draft queries, waiting for ETL pipelines to refresh, and waiting for static dashboards to render. This reactive model has gone past being just a bottleneck and now represents an existential risk for […]
The Apache Iceberg community recently finalised a new File Format API, scheduled for the upcoming 1.11.0 release. It is a strategic architectural shift that decouples the object model from the physical storage layout. The aim? To make file formats engine-agnostic, so Apache Iceberg can integrate with new formats without rewriting the core engine logic every […]
Data engineering teams are often defined by the gap between a business question and a verified answer. Requests languish in backlogs while engineering teams struggle with ETL pipelines and the technical debt of siloed warehouses. When reports finally arrive, the data is frequently stale, leading to debating the accuracy of the numbers rather than the […]
Last week I published a blog about some of the risks to consider when implementing Agentic AI in your workflows or organisation. But what about the other side of that coin, the benefits this revolutionary technology can bring about? Read on to learn several of the ways that Agentic AI is transforming how we work […]
Agentic AI offers several compelling benefits that are transforming how we work and solve problems. The key difference from traditional AI is agency, the agent’s ability to plan, use tools, take actions in sequence, and work toward goals rather than just responding to individual prompts. There are many upsides to embracing AI agents in your […]
Large Language Models (LLMs) are capable of understanding and generating language. However, they do not understand or process language in the same way that a person like you or I does. When reading text, humans construct meaning by processing the syntax of each sentence as it unfolds. This involves combining word definitions, using context from […]
Dremio has introduced several capabilities that inteliigently improve query performance across the data lakehouse. With minimal to no action from users, Dremio will reduce query latency, handle data maintenance tasks, and eliminate redundant compute jobs. This article is a summary of three of these performance management features. Read on to learn how reflections accelerate popular […]
Dremio is always striving to abstract away the physical concerns of data, whether the storage location, partitioning schema, or file size optimisation. Thanks to features such as Data Federation, Iceberg Clustering, and Autonomous Performance functionalities, Dremio users get highly-performant access to their data no matter where it lives. One of the components that delivers this […]
Caching dramatically reduces latency and computational costs by storing frequently accessed data closer to where it’s needed. Instead of repeated expensive operations – such as fetching from object storage, planning complex queries, or executing SQL – the data you need is provided in fast, local memory. To deliver on this, Dremio implements different layers of […]
Agentic AI is an artificial intelligence system that is designed to operate autonomously. With minimal human supervision it can be expected to make decisions and perform tasks with specifically trained agents. This is thanks in large part to Large Language Models (LLMs) which provide agentic AI with enhanced reasoning and the ability to understand context. […]
Reflections are a query acceleration functionality unique to Dremio, that work by minimising data processing times and reducing computational workloads. Debuting in the early days of Dremio, Reflections accelerate data lake queries by creating optimised Apache Iceberg data structures from file-based datasets, delivering orders-of-magnitude performance improvements. However, the game-changing aspect of this technology was not […]
A promise of self-service data platforms, such as the Data Lakehouse, is to democratise data. The idea is that they empower business users (BUs), those with little or no technical expertise, to access, prep, and analyse data for themselves. With the right platform and tools your subject matter experts can take work away from your […]
