Webinars

Serverless Cloud Data Lake with Spark for Serving Weather Data

The Weather Company (TWC) collects weather data across the globe at the rate of 34 million records per hour, and the TWC History on Demand (HoD) application serves that historical weather data to users via an API, averaging 600,000 requests per day. Users are increasingly consuming large quantities of historical data to train analytics models, and require efficient asynchronous APIs in addition to existing synchronous ones that use Elasticsearch.This session presents TWC’s architecture that uses a serverless cloud data lake running on top of Apache Spark and how that enables a highly elastic and economic way of serving weather history data. We will explain our concept of data skipping indexes that boosts performance by orders of magnitude compared to an out-of-the-box Spark setup, as well as significantly reducing cost. This enables TWC HoD to triple weather data coverage from land only to the entire globe, while at the same time reducing costs by an order of magnitude.We will also review serverless cloud data lake architecture in general and elaborate on the composition of serverless building blocks such as serverless storage, serverless ETL, serverless SQL and serverless data pipeline orchestration. In addition, we will review a set of major enhancements, including built-in geospatial and time series functions and a built-in multi-tenant Hive Metastore.Finally, we will highlight how TWC was able to adopt the serverless cloud data lake platform for new applications by rolling out a brand-new global data collection pipeline and data lake for COVID-19 data in just a few weeks.

Topics Covered

Apache Spark
Data Lake Storage

Speakers

Paula Ta-Shma

Paula Ta-Shma

Dr. Paula Ta-Shma is a Research Staff Member in the Cloud & Data Technologies group at IBM Research – Haifa and is responsible for a group of research efforts in the area of hybrid data, with a particular focus on high performance, secure and cost-efficient data stores and processing engines. She is particularly interested in performant SQL analytics over object storage and leads work on data skipping whose work is now integrated into multiple IBM products and services. Previously, she led projects in areas such as cloud storage infrastructure for IoT and continuous data protection. Prior to working at IBM, Dr. Ta-Shma worked at several companies on database management systems, including Informix Software Inc. where she worked on Apache Derby. She holds M.Sc. and PhD degrees in computer science from the Hebrew University of Jerusalem.

Torsten Steinbach

Torsten Steinbach

Torsten Steinbach has a long record working as a database architect. He led the IBM Db2 performance tooling and worked on the workload managers in IBM Netezza and IBM Db2. He also led the deep integration of machine learning into IBM’s RDBMS. Over the past few years, Torsten built from scratch IBM’s cloud data lake platform, which is heavily based on open source software such as Apache Spark and Apache Kafka.

Ready to Get Started? Here Are Some Resources to Help

Case Study

When E-Commerce Explodes – The More Data the More Dremio

read more

Webinars

Real-World Strategies to Optimize Data Platform Cost

read more
On-Demand webinar graphic

Webinars

Centralize Data Security Governance on your Open Data Lakehouse with Dremio & Privacera

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us