Auditing Your Data

At Nielsen, our data goes through a robust Kafka architecture into several ETLs to receive, transform, and store the data.

While we understood our ETLs’ workflow, we had no visibility into what parts of the data, if any, were lost or duplicated, and in which stage or stages of the workflow.

I will present the design process behind our Data Auditing system, Life Line, which uses Kafka, Avro, Spark, Lambda functions, and complex SQL queries. We’ll cover:

  • Avro Audit header
  • Auditing heartbeat – designing your metadata
  • Designing and optimizing your auditing table
  • Creating an alert-based monitoring system
  • Answering the most important question of all – is it the end of the day yet?


Simona Meriam

Simona Meriam

Simona Meriam is a Senior Data Engineer at Aidoc, where she specializes in research and development of solutions for big data infrastructures. In her previous position as a big data engineer at Nielsen, she researched and developed big data solutions using cutting-edge technologies such as Spark, Kafka, and Elasticsearch.

Ready to Get Started? Here Are Some Resources to Help

Case Study

When E-Commerce Explodes – The More Data the More Dremio

read more


Real-World Strategies to Optimize Data Platform Cost

read more
On-Demand webinar graphic


Centralize Data Security Governance on your Open Data Lakehouse with Dremio & Privacera

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us