File Formats Apache Parquet Subsurface LIVE Sessions

Data access restrictions, retention, and encryption-at-rest are fundamental security controls to achieve data privacy and compliance. This talk shows how we build and utilize open source Parquet’s finer-grained encryption feature to support all three controls in a unified way.

In particular, we will focus on the technical challenges of designing and applying encryption in a secure, reliable, and efficient manner for large-scale data. Those challenges include multiple access routes, performance overhead, handling the access denied, reliability, huge historical data, auto-onboarding, etc.

We will also share our experiences with recommended practices to manage the system in production at scale.