Webinars

Building an Efficient Data Pipeline for Data Intensive Workloads

Moving data through the pipeline in an efficient and predictable way is one of the most important aspects of modern data architecture, particularly when it comes to running data-intensive workloads such as IoT and machine learning in production. This talk breaks down the data pipeline and demonstrates how it can be improved with a modern transport mechanism that includes Apache Arrow Flight. This session details the architecture and key features of the Arrow Flight protocol and introduces an Arrow Flight Spark data source, showing how microservices can be built for and with Spark. Attendees will see a demo of a machine learning pipeline running in Spark with data microservices powered by Arrow Flight, highlighting how much faster and simpler the Flight interface makes this example pipeline.

Topics Covered

Apache Arrow Flight
Apache Spark
Data Lake Engines
Interfaces

Speakers

Ryan Murray

Ryan Murray

Ryan Murray is an open source Engineering Lead at Dremio. He previously served in the financial services industry doing everything from bond trader to data engineering lead. Ryan holds a PhD in theoretical physics and is an active open source contributor who dislikes it when data isn’t accessible in an organisation. He is passionate about making customers successful and self-sufficient, and still one day dreams of winning the Stanley Cup.

Ready to Get Started? Here Are Some Resources to Help

Case Study

When E-Commerce Explodes – The More Data the More Dremio

read more

Webinars

Real-World Strategies to Optimize Data Platform Cost

read more
On-Demand webinar graphic

Webinars

Centralize Data Security Governance on your Open Data Lakehouse with Dremio & Privacera

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us