From Discovering Data to Trusting Data

At Lyft, we have made our analysts and data scientists over 20% more productive by making it easier to discover data. Recently, we open sourced Amundsen and it’s now being used by ING, Square, Workday and many more.However, we ran into an interesting challenge. Not only is it now easy to discover good trusted data, it’s also easier to discover bad data that was previously hidden in the unforgotten nooks and crannies of the data lake. Consequently, we are now asking ourselves, “How can we recommend not just any data but trusted data to our users?”This talk will provide a quick overview of Amundsen and detail how we have tried both automated and curated metadata to showcase what’s trusted and what’s not trusted in Amundsen. It will dive deep into linking the Airflow DAG which produced the data (task-level lineage), linking what and how many dashboards are built from a given dataset (table-level lineage), as well as SLAs and historical landing times to give users a signal into what’s trusted.The talk will conclude with insights into current challenges and how we may solve them in the future.

Topics Covered

Subsurface Data Catalog by Dremio: Deep Insights.

Ready to Get Started? Here Are Some Resources to Help

Whitepaper Thumb


Dremio Upgrade Testing Framework

read more
Whitepaper Thumb


Operating Dremio Cloud Runbook

read more


Unlock the Power of a Data Lakehouse with Dremio Cloud

read more
get started

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now
demo on demand

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo
talk expert

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us

Ready to Get Started?

Bring your users closer to the data with organization-wide self-service analytics and lakehouse flexibility, scalability, and performance at a fraction of the cost. Run Dremio anywhere with self-managed software or Dremio Cloud.