Webinars

Streaming from an Iceberg Data Lake

The Apple AI/ML organization adopted Apache Iceberg data lake technology for numerous benefits. Iceberg can be leveraged as a streaming source. Streaming reads can further reduce processing delays from hours to minutes compared to periodically scheduled batch ETL jobs. In this talk, we will discuss how the Flink Iceberg source enables streaming reads from Iceberg tables. We will discuss the design of the source operator, focusing on the streaming read mode. We will compare the Kafka and Iceberg sources for streaming reads, and discuss how the Iceberg streaming source can power common stream processing use cases. Finally, we will present performance evaluation results. Download PDF

Speakers

Steven Wu

Steven Wu

Steven Wu is a software engineer at Apple. He is working on the stream processing infrastructure that powers data ingestion pipelines and streaming applications. Previously, he worked at Netflix where he helped build the real-time data infrastructure. He is passionate about building scalable distributed systems and empowering people with data.

Ready to Get Started? Here Are Some Resources to Help

Case Study

Case Study

Dremio Supports Moonfare’s High-Performance Culture with a High-Performance Lakehouse

Moonfare replaced a PostgreSQL-based data warehouse on Amazon Web Services (AWS) with a Dremio data lakehouse to offer data engineers, analysts and business users a high performance platform for business intelligence and predictive analytics empowering them to make better data-driven decisions.

read more

Case Study

Case Study: DB Cargo Gives Users the Green Light to All Data with Dremio

Deutsche Bahn Group (DB) is one of the world's leading mobility and logistics companies. The DB Cargo business unit manages DB's rail freight business.

read more
Case Study

Case Study

Case Study: Amazon Accelerates Supply Chain Decision Making with Dremio

Amazon's Supply Chain Finance Analytics team developed a new analytics architecture with Dremio to simplify ETL processes, accelerate queries, and provide analytics on a unified view of the data.

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us