Cross-Platform Lineage with OpenLineage

Data within today’s organizations has become increasingly distributed and heterogeneous. It can’t be contained within a single brain, a single team, or a single platform…but it still needs to be comprehensible, especially when something unexpected happens. Data lineage can help by tracing the relationships between datasets and providing a cohesive graph that places them in context.

Most data platforms have lineage capabilities, and can tell you how the datasets within their scope relate to one another. But very few of us can manage to operate within a single data platform. Fortunately, OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, and dbt. This provides a panoptical view that empowers a company to diagnose and address data quality and efficiency issues in real time across its entire ecosystem.

In this session, Michael Collado from Datakin will show how to trace data lineage across Apache Spark and dbt. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage.

Download PDF


Michael Collado

Michael Collado

Michael Collado is a seasoned developer with fifteen years of experience building websites, high-throughput services, and backend data infrastructure. Over the last decade, he has focused on big data analytics, catalog systems, and high-throughput data access. He has led multiple teams, including Amazon’s biggest clickstream data ingestion and analytics system and their primary website experimentation platform. He is currently a Staff Software Engineer at Datakin.

LinkedIn | Twitter

Ready to Get Started? Here Are Some Resources to Help

Case Study

When E-Commerce Explodes – The More Data the More Dremio

read more


Real-World Strategies to Optimize Data Platform Cost

read more
On-Demand webinar graphic


Centralize Data Security Governance on your Open Data Lakehouse with Dremio & Privacera

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us