Distributed Transactions on the Data Lake with Project Nessie

While database concepts like transactions, commits and rollbacks are necessary for traditional data warehousing workloads, they’re not sufficient for modern data platforms and data-driven companies. Project Nessie is a new open source metastore that builds on table formats such as Apache Iceberg and Delta Lake to deliver multi-table, multi-engine transactions. In this talk we will discuss the transactional model of Nessie and how it can help improve the ETL workflow. We will introduce the recently released Nessie Airflow provider and its use in multi-stage and complex workflows as an example of the power of Nessie transactions. We will finish with a demo and a discussion on the production readiness of Project Nessie.

Topics Covered

Metastores

Nessie

Table Formats

Speakers

Ryan Murray

Ryan Murray is an open source Engineering Lead at Dremio. He previously served in the financial services industry doing everything from bond trader to data engineering lead. Ryan holds a PhD in theoretical physics and is an active open source contributor who dislikes it when data isn’t accessible in an organisation. He is passionate about making customers successful and self-sufficient, and still one day dreams of winning the Stanley Cup.

Ready to Get Started? Here Are Some Resources to Help

Webinars

Smart Data – Smart Factory with Octotronic and Dremio

Guides

What Is a Data Lakehouse?

The data lakehouse is a new architecture that combines the best parts of data lakes and data warehouses. Learn more about the data lakehouse and its key advantages.

Whitepaper

Simplifying Data Mesh for Self-Service Analytics on an Open Data Lakehouse

The adoption of data mesh as a decentralized data management approach has become popular in recent years, helping teams overcome challenges associated with centralized data architecture.