Functional Data Engineering – A Set of Best Practices

Batch data processing (also known as ETL)  is time-consuming, brittle and often unrewarding. Not only that, it’s hard to operate, evolve and troubleshoot.In this talk, we’ll discuss the functional programming paradigm and explore how applying it to data engineering can bring a lot of clarity to the process. It helps solve some of the inherent problems of ETL, leads to more manageable and maintainable workloads and helps to implement reproducible and scalable practices. It empowers data teams to tackle larger problems and push the boundaries of what’s possible.