Cross-Platform Lineage with OpenLineage
Data within today’s organizations has become increasingly distributed and heterogeneous. It can’t be contained within a single brain, a single team, or a single platform…but it still needs to be comprehensible, especially when something unexpected happens. Data lineage can help by tracing the relationships between datasets and providing a cohesive graph that places them in context.
Most data platforms have lineage capabilities, and can tell you how the datasets within their scope relate to one another. But very few of us can manage to operate within a single data platform. Fortunately, OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, and dbt. This provides a panoptical view that empowers a company to diagnose and address data quality and efficiency issues in real time across its entire ecosystem.
In this session, Michael Collado from Datakin will show how to trace data lineage across Apache Spark and dbt. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage.
