Scaling Up Apache Airflow to Enterprise Level

Rabobank the Netherlands is building an enterprise-grade data mesh. Our team was tasked with implementing Apache Airflow as the de facto orchestration tool. For hundreds of teams.

Looking back, we can finally say we did it. Airflow is now hosting over 50 teams, running smoothly, linking data delivery streams with data consumption streams. However, we did not get to this point without making our fair share of mistakes.

This session will cover every step of our journey, including: dealing with user team proficiency (or the lack thereof), struggling with the Kubernetes Executor, trying our hand at zero-downtime deployments, getting our scaling right, and fighting with the PostgreSQL database backend.

Download PDF

Speakers

Martijn Beenker

Martijn Beenker is a Senior Data/Platform Engineer at Avanade. He cares about team happiness and believes “Automation enables recreation.”

Ready to Get Started? Here Are Some Resources to Help

Using Data Mesh to Advance Distributed Data Access, Agility and Governance

Join this live fireside chat to learn about using Data Mesh to Advance Distributed Data Access, Agility and Governance.

Webinars

Smart Data – Smart Factory with Octotronic and Dremio

Guides

What Is a Data Lakehouse?

The data lakehouse is a new architecture that combines the best parts of data lakes and data warehouses. Learn more about the data lakehouse and its key advantages.