Streaming from an Iceberg Data Lake

The Apple AI/ML organization adopted Apache Iceberg data lake technology for numerous benefits. Iceberg can be leveraged as a streaming source. Streaming reads can further reduce processing delays from hours to minutes compared to periodically scheduled batch ETL jobs.

In this talk, we will discuss how the Flink Iceberg source enables streaming reads from Iceberg tables. We will discuss the design of the source operator, focusing on the streaming read mode. We will compare the Kafka and Iceberg sources for streaming reads, and discuss how the Iceberg streaming source can power common stream processing use cases. Finally, we will present performance evaluation results.

Download PDF

Ready to Get Started? Here Are Some Resources to Help


2024 State of the Data Lakehouse

Benchmark your organization with Dremio's State of the Data Lakehouse Survey Report!

read more

Expert Panel Discussion – Data Integration Trends and Best Practices Webinar

TDWI senior research director James Kobielus will engage data industry experts in an in-depth discussion of data integration trends and best practices

read more
OctotronicDremio Smart Data Smart Factory 1


Smart Data – Smart Factory with Octotronic and Dremio

Wie bringt ein Data Lake House die Smart Factory auf ein neues Level?

read more
get started

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now
demo on demand

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo
talk expert

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us