A Git-Like Experience for Data Lakes

While traditional data warehouse concepts like transactions, commits and rollbacks are necessary in a SQL context, they’re not sufficient for modern data platforms, which involve many users and data flows. A new open source project, Project Nessie, decouples transactions and makes distributed transactions a reality by building on the capabilities of table formats like Apache Iceberg and Delta Lake to provide Git-like semantics for data lakes. By using versioning concepts, users can work in an entirely new way, experimenting or preparing data without impacting the live view of the data, opening a whole world of possibilities for true DataOps on the data lake. At the same time, they can leverage classic data warehousing transactional concepts in their data lake including three levels of isolation across all tables and the classic START…COMMIT primitives.