Data Lineage and Observability with Marquez
Data is increasingly becoming core to many products. Whether to improve recommendations for users, get insights on how they use the product, or use machine learning to improve the experience. This creates a critical need for understanding how data flows through systems. Data pipelines must be auditable, reliable and run on time.Tracking lineage and metadata is the underlying foundation that enables many use cases related to data. It provides understanding of dependencies between many teams consuming and producing data and how constant changes impact them. It enables governance and compliance and generally helps you keep your data running.Marquez is an open source project, and part of the LF AI, which instruments data pipelines to collect lineage and metadata and enable those use cases. It provides context by making visible dependencies across organisations and technologies and enables lineage governance and discovery.
Julien Le Dem
Julien Le Dem is the Chief Architect of Astronomer and Co-Founder of Datakin. He co-created Apache Parquet and is involved in several open source projects including OpenLineage, Marquez (LFAI&Data), Apache Arrow, Apache Iceberg, and others. Previously, he was a senior principal at WeWork, a principal architect at Dremio, a tech lead for Twitter’s data processing tools, where he obtained a two-character Twitter handle (@J_), and a principal engineer and tech lead working on content platforms at Yahoo, where he received his Hadoop initiation. His French accent makes his talks particularly attractive.