Data Virtualization seems promising. But does it scale with your data and BI needs?

However, as the amount of data, users, and applications grow in an organization, the first problem encountered is performance. Users aren’t able to get answers to their questions nearly as fast as they need to be effective. These performance problems result in users asking fewer questions or even being unable to wait for an answer at times. The result is poor decisions based on hunches or guesses lacking data.

The causes of these performance problems fall into one or more of the following categories:

• The source system is unable to send the data to the data virtualization platform fast enough at runtime, whether because of source system load or the design of the source’s storage system.
• The network speed between the data virtualization platform and the source system isn’t fast enough to transfer the data at runtime.
• The protocol used, generally JDBC, isn’t able to transmit the data from the source system to the data virtualization platform fast enough at runtime.
• The data is being transferred from the source system to the data virtualization platform over a single connection, i.e. serially.
• At a certain scale, the sheer amount of data that needs to be processed at runtime makes it impossible to do interactive analytics, simply due to the physics of reading that much data.

Because all of these problems stem from transferring the data at runtime, the solution to this performance problem is generally to transfer the data ahead of time on a case-by-case basis by making copies of some datasets into a Data Warehouse and connecting the Data Warehouse to the Data Virtualization tool, too. This method also addresses needs that arise for removing the analytic workload from the operational systems, to ensure the operational system can serve its primary function properly.

However, as the data, user, and application scale continues to grow, generally one Data Warehouse for the whole organization ends up being insufficient, whether for flexibility, cost, complexity, or operational reasons. Typically at that point, each business unit gets their own Data Warehouse or Data Mart. Once again, these Data Warehouses are connected to the Data Virtualization platform, and data consumers have self-service access to a wide range of data.

The performance problems of transferring data at runtime when joining between data in two Data Warehouses still apply for exactly the same reasons as before. So, when needing to join data at this scale between the two systems, this is done via IT request and creating an ETL pipeline to copy the data into the destination Data Warehouse.

Augmenting Data Virtualization with data copies to address the performance problems now causes a whole new set of problems at this scale:

• Lack of self-service 
• Data engineering overhead
• Slow turnaround time for requests
• Data drift 
• Regulatory compliance issues
• Infrastructure cost

The workaround of creating data copies to address performance problems associated with Data Virtualization is a big topic worth its own dedicated page. For more information, see this whitepaper discussing the unexpected cost of data copies.

Data Virtualization was supposed to help solve some of these problems. Instead, when used at scale, it exacerbates the problems and introduces new issues you didn’t have before.

All of these problems are downstream effects of a lack of performance in the Data Virtualization approach. So, if we can solve the performance problems, we can avoid the workarounds that cause these downstream issues. 

Let’s recall the causes of the performance problems of Data Virtualization in the first place, and see how we can address them:

Data Virtualization Performance Problems	Problem Mitigation
The source system is unable to send the data to the DV platform fast enough at runtime because of source system load or the design of the source’s storage system.	The processing engine fulfilling user requests needs to have direct access to the storage and data.
The network speed between the DV platform and the source system isn’t fast enough to transfer the data at runtime.	We need a performant network between the processing engine and the storage engine. When the data gets too large to be feasibly addressed via network throughput, we need to get the data closer via caching to reduce the network bandwidth needed or via pre-computation to reduce the amount of data needed to be transferred.
The protocol used, generally JDBC, isn’t able to transmit the data from the source system to the Data Virtualization platform fast enough at runtime.	The processing engine fulfilling user requests needs to be able to access the data stored in columnar format using a protocol that doesn’t needlessly serialize the columnar data to a row format, just for it to be deserialized back to columnar format in the processing engine, i.e., it needs to be able to get the data for processing while the data stays in columnar format.
The data is being transferred from the source system to the Data Virtualization platform over a single connection, i.e., serially.	The processing engine fulfilling user requests needs to be able to access the storage system with a high level of parallelism.
At a certain scale, the sheer amount of data that needs to be processed at runtime makes it impossible to do interactive analytics, simply due to the physics of reading that much data.	The processing engine needs to provide the ability to do various pre-computations of the data, to reduce the amount of data needed to be processed at runtime.

If we look at the above requirements all together, the conclusion is that, in order to meet our business analytic goals, the majority of the data needs to be physically moved out of the source systems and into a centralized platform.

Data teams require:

• A semantic/logical view of all the data to your end users to enable self-service.
• Interactivity on that logical view, without the need for pointing users to other marts or areas.

It may be thought one could achieve this with any massively parallel processing (MPP) SQL query engine that can query data lakehouses directly? Not exactly…. Performance is only part of the equation when thinking about solving for big data analytics at scale.

The ease of optimizations (Dremio Data Reflections) allow data professionals to unlock data consumers to get the most value from their data without proliferating copies of data everywhere. Only when an engine nails both of these issues together will it have the desired impact on your organization's data challenges.

Dremio offers very simple optimizations to complex workloads (reflections). The downstream effects are massive, e.g., no app code changes, easily meet SLAs. The process of data exploration through  productionalizing a data application is on a different level with a SQL platform like Dremio. Users are able to find data, define their logic and optimize the performance without moving any data out of the data lakehouse. 

Other tools are able to serve as the SQL layer but require many other steps and potentially multiple copies of the data for users to observe the performance they want - especially from BI style workloads.

Productionalizing interactive reports/dashboards

These optimizations and incredible performance gains require no changes at the application level, i.e., BI tools, dashboards, report logic, etc. This makes it extremely simple for a wide range of users to benefit from these optimizations taking place at the query engine level.

There are several advantages to forming your data foundation on a data lakehouse via open formats (parquet/iceberg). These benefits are magnified when you introduce the most advanced SQL engine on top (Dremio).  Set your team up for current and future data success...try Dremio today!