Everything was going great, your data was in your data lake, queries were fast and the SREs were happy. But then things started to slow down. Queries took longer, even specific queries which used to be fast now take a long time. The culprit? Small and unorganized files.
The solution? Apache Iceberg’s RewriteDatafile action. In this talk, Russell Spitzer will dive into how RewriteDataFiles can 1) right-size your files, merging small files and splitting large ones, ensuring that no time is waisted in query planning or in opening files; and 2) reorganize the data within your files, supporting hierarchal sort and multidimensional ordering algorithms, enabling you to make sure your data is optimally set out for your queries. With these two capabilities, any table can be kept at peak performance regardless of ingestion patterns and table size.Russell-Spitzer-ManageDataFiles_Final
Russell Spitzer received his Ph.D from UCSF after performing a lot of comparisons of protein binding sites. Following that he joined DataStax to get more involved in the Big Data / Distributed Computing Scene. After spending the majority of his time developing integrations of Cassandra with Hadoop, Spark, Solr and a bunch of open source technologies, he has moved on to become a member of the Apache Iceberg PMC.