Hadoop Migration

Customers with Successful Migrations

Why Migrate and Modernize Your Hadoop Data Lake with Dremio?

Hadoop is slow, costly, and time-consuming to manage. At one time, the advantages of Hadoop overcame these drawbacks and their hidden costs. But over the years many enterprises have found that Hadoop is cheap to provision and expensive to maintain. This impacts an organization’s ability to efficiently provide data access for end users.

Hadoop customers are now migrating to object storage, such as Amazon S3, and building their data lakehouse with Dremio for sub-second query performance, and a unified access layer for governed self-service analytics.

Benefit 1

Sub-Second Query Performance and 10x Better Price / Performance

Dremio delivers sub-second queries to your dashboards that run directly on cloud data lake. Build interactive and ad-hoc queries without worrying about the underlying table structure.

interactive screenshot showing how to set your sub-second queries to your dashboards that run directly on cloud data lake

screenshot of space privileges check list

Benefit 2

Governed Self-Service Analytics

Often, due to Hadoop performance issues, IT locks down data access and only provides curated datasets to analysts through a separate data warehouse. The lack of self-serve access hurts business agility and time-to-insight while burdening engineering teams with mundane query and access requests. With Dremio, data consumers can use a modern user interface for self-service data curation and access. Easily govern data with fine-grained access controls and provide full visibility into your data with Dremio's data lineage capabilities.

Learn how Dremio secures your data with enterprise governance. ->

Benefit 3

Unified View Across All Your Data

Most organizations have data spread across databases, data warehouses, and data lakes. Siloed data makes it difficult to provide business users with data access if there is never a single source of truth. Dremio federates queries across numerous cloud and on-premise data sources improving data access and agility. Build reliable data products and make metrics and business logic consistent across all your downstream apps.

federated computational governance chart

"This is a huge advantage that Dremio offers. When you decide to change the infrastructure of your data you don't need to impact your reporting layer. Having an independent query engine enables us to play with different solutions, and we can optimize for cost, time to market and easily move data back and forth"

Lotar Schin

Big Data Team Lead | OTP Bank

Read OTP Bank’s case study (pdf)

Migrating from Hadoop to an Open Data Lakehouse

We've taken part in the Hadoop migration and modernization journeys of many of the largest companies in the world, including TransUnion, The Hartford, and more. Migrating from Hadoop takes time and should be done with minimal impact on production. We've found that customers that start their migration by modernizing with Dremio, typically provide the best experience to their business users and their organization. We recommend three steps in the journey, (1) modernize the query engine and provide self-service, (2) migrate off Hadoop, and (3) create your open data lakehouse.

Step 1

Modernize Query Engine and Provide Self-Service

The query engines provided with Hadoop distributions can't meet the needs of business users and analysts. Enhance the usability of your HDFS cluster and achieve more than 10x query performance over Hive and Impala with Dremio.

Moving to Dremio also allows organizations to connect and federate queries across multiple other data sources for self-service analytics. Easily democratize data products across domains with Dremio as the unified access layer.

Step 2

Migrate off Hadoop

With the business access layer established for data consumers, the move to object storage, such as Amazon S3, can occur without business users even noticing the migration. Customers can use Dremio software on top of their existing HDFS cluster and Dremio in the cloud for data migrated to cloud object storage. We provide a connector between the Dremio instances to ensure this hybrid experience is seamless for the business. Dremio provides a flexible deployment model to accommodate your specific object storage requirements, enabling a smooth and secure migration from HDFS.

Step 3

Create an Open Data Lakehouse

Simplify your architecture with the data lakehouse and take advantage of the latest innovations in open source standards such as Apache Iceberg. Get sub-second performance over the data lake, with warehouse features like DML, schema evolution, time travel, and more. Dremio is built for the latest Iceberg functionality and makes lakehouse management easy with data catalog, table optimization, and data-as-code capabilities.

Learn how to modernize Hive to the Data Lakehouse with Apache Iceberg.