Apache Iceberg – An Architectural Look Under the Covers

   
  • Jason HughesDirector of Product Management, Dremio

Session Abstract

Data Lakes have been built with a desire to democratize data – to allow more and more people, tools, and applications to make use of data. A key capability needed to achieve it is hiding the complexity of underlying data structures and physical data storage from users. The de-facto standard has been the Hive table format, released by Facebook in 2009 that addresses some of these problems, but falls short at data, user, and application scale. So what is the answer? Apache Iceberg. Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS.Join Jason Hughes, Technical Director at Dremio, for this webinar to learn the architectural details of why the Hive table format falls short and why the Iceberg table format resolves them, as well as the benefits that stem from Iceberg’s approach.You will learn:

  • The issues that arise when using the Hive table format at scale, and why we need a new table format
  • How a straightforward, elegant change in table format structure has enormous positive effects
  • The underlying architecture of an Apache Iceberg table, how a query against an Iceberg table works, and how the table’s underlying structure changes as CRUD operations are done on it
  • The resulting benefits of this architectural design

Ready to Get Started? Here Are Some Resources to Help

Analyst Report

DZone Data Pipelines Trend Report

DZone has released their Data Pipelines Trend Report. It is a survey of software developers, architects, platform engineers, and IT professionals to understand the challenges and potential solutions around ingesting, processing, and leveraging data.

read more

Webinars

How Enel Group Built a Data Mesh Architecture with Dremio and AgileLab

In this webinar, learn how Enel Group worked with Agile Lab to implement Dremio as a data mesh solution for providing broad access to a unified view of their data, and how they use that architecture to enable a multitude of use cases.

read more

Webinars

The Digital Bank: Building a data-driven customer experience on the open lakehouse

Learn how direct access to analytics on Amazon S3 with Dremio can help banking and finance firms achieve a consistent and data-driven customer experience.

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

Watch Demo

Not ready to get started today? See the platform in action.

Check Out Demo