Editor’s note, September 2026. This post was published in February 2022 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current catalog layer works, see Choosing an Apache Iceberg Catalog.
The most exciting innovation happening in the open lakehouse ecosystem is being driven by the open source software (OSS) community.
These technologies liberate access to data from proprietary walled gardens, enable high-speed reads and writes, and enable robust security controls to comply with modern regulatory demands.
Some of those key technologies include Apache Arrow, Apache Iceberg and Project Nessie. If you’re working as a data engineer or data architect, or play another role on a data team, these are the technologies that are shaping the industry. How can you expand your knowledge of them and deepen your expertise? These talks, given by the people who built and ran these systems, remain the clearest introductions to each one.
Apache Parquet provides a columnar file format that is quick to read and query, but what happens after the file is loaded into memory? This is what Apache Arrow aims to answer by being an in-memory columnar format to maximize the speed of query processing in memory. The Arrow Flight connector allows any JDBC compatible data store to take advantage of Arrow.
Here is a session from Subsurface LIVE 2020. Check it out to learn more about the role Apache Arrow and Arrow Flight play in disrupting previous approaches to creating data services that transport large datasets. Watch it to learn about the technical details of why the Arrow protocol is an attractive choice as well as specific examples of where Arrow has been employed for better performance and resource efficiency.
Apache Iceberg is an open table format creating a new paradigm in defining tables on the data lake. Not only does it separate storage of the data from the metadata that tracks the table but it tracks the metadata across a tree of files. This separation of metadata concerns allows query planning to be blazing-fast while enabling features like hidden partitioning, partition/schema evolution, time-travel, version rollback, safe update/delete transactions, and safe concurrent writing.
Watch this presentation to learn about the challenges and motivations behind building Apache Iceberg as the next generation of big data analytical storage. You’ll hear about the current state and roadmap for production deployments. Lastly, you’ll learn about the future of automated storage optimization and compute-based enhancements built on machine learning algorithms at Netflix.
This talk from Iceberg PMC member Anton Okolnychyi describes how to maintain Iceberg tables in their optimal shapes while running at petabyte scale. You’ll learn how to efficiently perform metadata and data compaction on Iceberg tables with millions of files without any impact on concurrent readers and writers.
In this talk, Iceberg co-creator Ryan Blue introduces the use cases for Apache Iceberg tables that weren’t expected when it was created and explains the details so you can use Iceberg for similar cases.
While Apache Iceberg allows several new possibilities at the table level, Project Nessie unlocks a git-like experience at the lake level. Using Project Nessie you can create branches to isolate work on several tables in a catalog without affecting data consumers querying the main branch, which can then be merged as a multi-table transaction when work on the branch is complete. This creates truly new possibilities for effective data collaboration and workflows.
Since this talk was recorded the catalog layer has developed further. Apache Polaris, which Dremio co-created and donated to the Apache Software Foundation, is now a Top-Level Project implementing the Iceberg REST Catalog specification. Nessie’s git-like branching and Polaris’s open catalog approach address related but distinct problems, and both remain in use today.
While database concepts like transactions, commits and rollbacks are necessary for traditional data warehousing workloads, they’re not sufficient for modern data platforms and data-driven companies. Project Nessie is a new open-source metastore that builds on table formats such as Apache Iceberg and Delta Lake to deliver multi-table, multi-engine transactions. In this talk, you’ll learn about the transactional model of Nessie and how it can help improve the ETL workflow.
For a deeper treatment of the table format at the centre of these talks, download Apache Iceberg: The Definitive Guide by Tomer Shiran, Jason Hughes, and Alex Merced, free from Dremio.
For structured, hands-on versions of this material, explore the free lakehouse and Iceberg courses at Dremio University.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Editor’s note, September 2026. This post was published in September 2023 and some product details have changed since. Nessie is still available and self-deployable under the Apache-2.0 licence, and it remains the clearest implementation of catalog-level branching. It is not an Apache Software Foundation project, and its development has slowed considerably. For how the current […]
With its capabilities in on-prem to cloud migration, data warehouse offload, data virtualization, upgrading data lakes and lakehouses, and building customer-facing analytics applications, Dremio provides the tools and functionalities to streamline operations and unlock the full potential of data assets.
This is Part 15, the final article of a 15-part Apache Iceberg Masterclass. Part 14 covered hands-on Dremio Cloud. This article covers the three migration strategies and how to execute a zero-downtime migration using the view swap pattern. Most organizations do not start with Iceberg. They have years of data in Hive tables, data warehouses, CSV files, databases, […]