The Solution – Cloud and Dremio
Umberto Misso: "During the data lake implementation, we became aware of Dremio and had already checked out the data lake engine as the ideal tool for viewing our raw data in the data lake."
After a kick-off with Dremio and a short but intense proof-of-concept phase, the CRM team was certain: Dremio will be the right analytics environment for the data lake they built.
The flexible and fast integration of surrounding microservice-based data pools is already an essential advantage in the data lake context. But having a feature that allows working directly with the data based on the Parquet files in the data lake is an even greater one.
"Where we previously had to create Data Factory pipelines and transformation processes, Dremio provides data exploration with just a few clicks. In order to set up analytic structures for the in-house analysts and to guarantee a matching restricted access, we created a structure within the Dremio Engine based on combined Physical Data Sets (PDS as a summary of underlying Parquet files) and Visual Data Sets (virtually merged data) and paired it with a role base authorization concept that allows us to control country access", explains Umberto Misso.
From e-commerce (SAP Hybris V6) to POS, legacy and other on-premise systems, all data now flows into an Azure Data Lake where all CRM-relevant information can be analyzed like on a gigantic hard drive. While data ingestion is handled by Azure Databricks and Data Factory, processing in Apache Spark ensures lightning-fast results.
Another component of the data lake is a supply line created with CampaignX, a proprietary campaign management solution the Douglas IT developed back in 2011. The solution, which has been tailored to the business and the company's requirements, is still used today by operational marketing. Integrating the Dremio layer with the campaign management solution is part of the internal roadmap.
As a powerful data lake engine, Dremio is the central platform for all exploratory and ad hoc analytic queries and also serves as a data validation tool during data layer development.
For the implementation of the new data lake engine Dremio consultants and Douglas IT worked hand in hand. As the experienced data experts had already prepared a lot, users could quickly leverage the new cloud architecture.
The Results
The CRM team sees itself as service provider for the business units and uses its expertise to support them. Before these specialists will implement something in technology they will always consider the analytical aspect, too. This is why Umberto Misso and his colleagues were so keen to use Dremio, because the fast exploratory access to the underlying data empowers their "customers" to better understand and use their data.
Quality analytics instead of ping-pong effect
Unlike the CRM teams, many users have no idea what's possible with their data. To avoid toing and froing, IT and business use their daily coordination meetings to discuss requirements in detail. Then Dremio prepares the data in exactly the way marketers need it for high-quality analyses. Subsequently, the functions of the virtual datasets can be transferred to the data factory pipelines. Data transformations are then moved to the underlying layer, thus lightening the load of the analytical infrastructure.
Single point of truth
All data is held centrally and delivered via Dremio. There is no more uncertainty about which data is correct or the most up-to-date. Furthermore, the IT team has now the flexibility to respond quickly to new requirements and can even create new reports for ad hoc queries in a flash. The tedious collection of data from a variety of source systems has turned into an easy delivery. Another plus: Data loaded into the data lake in near-time can also be updated in near-time.
Plug & play for analysts' favorite tools
Interfaces like ODBC and JDBC enable power users to access Dremio data either through the Dremio user interface or with external tools like SQL clients or R. In addition, Dremio offers a variety of native connectors that deliver data to Tableau, Power BI or other tools for super-fast visual BI analysis and dashboards.
A new dimension of CRM – customer lifecycle
From receipt data to the last online transaction, every purchase was and is recorded. With Dremio, marketers also get access to historical data and can analyze and combine it with current information. Therefore, they can not only identify KPIs like total sales by customer but also improve the effectiveness of cross–selling activities.
Speed and ease-of-use – time to data
In the past, building a data pipeline took 3 days. With Dremio this now takes only 3 minutes and from an idea for an analysis to the required data needs only a few clicks.
Security and governance
With Dremio users can only see the data they are allowed to see. Access rights can be restricted to specific datasets so that, for example, users can only access information from their own country. Dremio ensures adherence to internal governance guidelines as well as GDPR compliance and it is of course possible to anonymize data.
