Tracking & Triggering Pattern with Spark Stateful Streaming

Inside Adobe Experience Platform we noticed we needed to track actions happening at the control plane level and act upon them at lower levels like data lake, ingestion processes, etc. Using Apache Spark Stateful Streaming we’ve been able to create services that act by starting processes like compacting data, consolidating data, and cleaning data, minimizing processing time while keeping everything under defined SLAs. This talk presents a pattern that we’ve been using in production for the last two to three years inside Adobe Experience Platform in multiple services and with no high-severity on-call interventions and minimal-to-none operational costs on high throughput ingestion flows.

Topics Covered

Apache Spark
Data Lake Engines

Ready to Get Started? Here Are Some Resources to Help

Using Data Mesh to Advance Distributed Data Access, Agility and Governance

Join this live fireside chat to learn about using Data Mesh to Advance Distributed Data Access, Agility and Governance.

read more


Smart Data – Smart Factory with Octotronic and Dremio

read more


What Is a Data Lakehouse?

The data lakehouse is a new architecture that combines the best parts of data lakes and data warehouses. Learn more about the data lakehouse and its key advantages.

read more

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us