Orchestrating Data Validation Workflows with Prefect

As data pipelines become increasingly complex and interconnected, workflow management systems are being used to schedule and monitor tasks. Prefect is an open source workflow management system designed for large-scale data processes.In this talk, we’ll show how to get started with Prefect to orchestrate data validation workflows to preserve data integrity before it’s used in downstream processes.

Speakers

Kevin Kho

Kevin Kho is an Open Source Community Engineer at Prefect, an open source workflow orchestration management system. Previously, he was a data scientist at Paylocity, where he worked on adding machine learning features to their Human Capital Management (HCM) suite.

Outside of work, he is a contributor for Fugue, an abstraction layer that lets users port Python and SQL code to Pandas, Spark and Dask. He also organizes the Orlando Machine Learning and Data Science Meetup.

Ready to Get Started? Here Are Some Resources to Help

Using Data Mesh to Advance Distributed Data Access, Agility and Governance

Join this live fireside chat to learn about using Data Mesh to Advance Distributed Data Access, Agility and Governance.

Webinars

Smart Data – Smart Factory with Octotronic and Dremio

Guides

What Is a Data Lakehouse?

The data lakehouse is a new architecture that combines the best parts of data lakes and data warehouses. Learn more about the data lakehouse and its key advantages.