Covering Indexes in the Data Lake with Hyperspace
How can ‘Data processing and querying in the Data Lake at scale’ can be improved? Given that our Data Lake supports hundreds of customers with wide (thousands of columns), heavy (10 and 100 TB of data), and fast changing (thousands of appends/deletes/etc) datasets, we need reliability and scalability for a fast changing data environment. At Adobe Experience Platform we try to use the right tools for the right use case. So for our Data Lake, we chose to Apache Iceberg as our core foundation and Hyperspace as our indexing subsystem. In this session, Andrei Ionescu explains how Hyperspace integrates with Iceberg and other formats to speedup the processing time while keeping the data consistent and performant. Watch to the end of the talk to see a demo of Apache Iceberg and Hyperspace.
Andrei Ionescu is a Senior Software Engineer with Adobe, and he is part of Adobe Experience Platform’s Data Lake team, specializing in big data and distributed systems with Scala, Java, Spark, and Kafka. At Adobe, he is mainly contributing to ingestion and data Lake projects, while on open source he is contributing to Hyperspace and Apache Iceberg.