Cross-Platform Lineage with OpenLineage

Data within today’s organizations has become increasingly distributed and heterogeneous. It can’t be contained within a single brain, a single team, or a single platform…but it still needs to be comprehensible, especially when something unexpected happens. Data lineage can help by tracing the relationships between datasets and providing a cohesive graph that places them in context.

Most data platforms have lineage capabilities, and can tell you how the datasets within their scope relate to one another. But very few of us can manage to operate within a single data platform. Fortunately, OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, and dbt. This provides a panoptical view that empowers a company to diagnose and address data quality and efficiency issues in real time across its entire ecosystem.

In this session, Michael Collado from Datakin will show how to trace data lineage across Apache Spark and dbt. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage.

Download PDF

Ready to Get Started? Here Are Some Resources to Help


Dremio’s Well-Architected Framework

read more
Whitepaper Thumb


Harness Snowflake Data’s Full Potential with Dremio

read more
Whitepaper Thumb


Simplifying Data Mesh for Self-Service Analytics on an Open Data Lakehouse

read more
get started

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now
demo on demand

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo
talk expert

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us

Ready to Get Started?

Bring your users closer to the data with organization-wide self-service analytics and lakehouse flexibility, scalability, and performance at a fraction of the cost. Run Dremio anywhere with self-managed software or Dremio Cloud.