Bloomberg’s Enterprise Data business has accumulated petabytes of historical financial data by taking point-in-time “snapshots” of financial entities and their attributes over four decades. Historical financial data is critical in backtesting models, evaluating risk, regulatory reporting, evaluating data quality, and more. Our Enterprise Data Lake engineering group ingested all historical text files (plus the ongoing snapshots that continue flowing in) into Apache Iceberg tables. This talk will include an overview of the challenges our organization needed to address, the open source architecture/tools we chose (Iceberg, Trino, etc.), and the impact this initiative has had on our business.
Many agencies today are struggling not only with managing the scale and complexity of cyber data but also with extracting actionable insights from that data. With new data retention regulations, such as M-21-31, compounding this problem further, agencies need a next-generation solution to address these challenges.
read more
