Webinars
March 3, 2022
Streaming from an Iceberg Data Lake
The Apple AI/ML organization adopted Apache Iceberg data lake technology for numerous benefits. Iceberg can be leveraged as a streaming source. Streaming reads can further reduce processing delays from hours to minutes compared to periodically scheduled batch ETL jobs.
In this talk, we will discuss how the Flink Iceberg source enables streaming reads from Iceberg tables. We will discuss the design of the source operator, focusing on the streaming read mode. We will compare the Kafka and Iceberg sources for streaming reads, and discuss how the Iceberg streaming source can power common stream processing use cases. Finally, we will present performance evaluation results.
Speakers
Steven Wu
Steven Wu is a software engineer at Apple. He is working on the stream processing infrastructure that powers data ingestion pipelines and streaming applications. Previously, he worked at Netflix where he helped build the real-time data infrastructure. He is passionate about building scalable distributed systems and empowering people with data.
