8 Tools For Ingesting Data Into Apache Iceberg

Data platforms increasingly migrate to data lakehouses, particularly those built on Apache Iceberg tables. Once you've selected the catalog to track your Apache Iceberg tables, the next critical decision is determining how you'll ingest your data—in batch or streaming—into those tables. In this article, we'll explore eight tools that enable data ingestion into Iceberg and resources that provide hands-on guidance for using these tools.

Data Lakehouse Platforms

Data Lakehouse platforms are designed specifically for implementing data lakehouses. They offer tools for querying, ingesting, managing, and governing data within the lakehouse, among other capabilities.

Dremio

Dremio is a data lakehouse platform that offers significant value to those looking to elevate their data lake into a fully-fledged data lakehouse across three key categories:

• Unified Analytics: Dremio enables you to connect your data lake, databases, and data warehouses, both in the cloud and on-premises. This allows you to organize, model, and govern all your data in a unified environment.
• SQL Query Engine: Dremio features a built-in query engine that delivers industry-leading price/performance. It allows you to federate queries across all connected sources and supports fine-grained access controls, enabling row—and column-level access rules.
• Lakehouse Management: Dremio includes an integrated lakehouse catalog with Git-like semantics at the catalog level, providing robust tracking of your Apache Iceberg tables. It offers automated management features to optimize and maintain your lakehouse, so you don't have to worry about it. Additionally, Dremio connects with various Apache Iceberg catalogs, making it a cornerstone of any Iceberg-based lakehouse.

Articles About Ingesting Data into Iceberg with Dremio:

• Ingesting Data into Iceberg with Dremio
• Ingesting Data from Postgres
• Ingesting Data from MongoDB
• Ingesting Data from SQLServer
• Ingesting Data from MySQL
• Ingesting Data from Apache Druid
• Ingesting Data from JSON, CSV and Parquet
• Ingesting Data using DBT, Git-for-Data and Dremio

Open Source Tools

Numerous open-source tools are available to help ingest data into Apache Iceberg. In this section, we'll highlight a few of these tools and direct you to articles that guide how to use them with your data.

Apache Spark

Apache Spark is a well-known name in open-source data engineering. It offers robust capabilities for handling both batch and streaming workloads.

Articles About Ingesting Data into Iceberg with Apache Spark:

• Getting Started with Apache Spark and Apache Iceberg
• Getting Started with Apache Spark, Nessie and Apache Iceberg

Apache Flink

Apache Flink is a stateful stream processing tool designed to ingest streaming data from various sources into any destination.

Articles About Ingesting Data into Iceberg with Apache Flink:

• Using Apache Flink with Apache Iceberg and Flink

Kafka Connect

Kafka Connect is a data integration tool that facilitates ingesting data from an Apache Kafka topic into a specified destination.

Articles About Ingesting Data into Iceberg with Kafka Connect:

• Ingesting Data into Iceberg with Kafka Connect and Nessie

Data Ingestion/Integration Platforms

Upsolver

Upsolver is a cloud-native data ingestion platform optimized for handling high-volume streaming data and efficiently ingesting it into destinations like Apache Iceberg.

Articles About Ingesting Data into Iceberg with Upsolver:

• Ingesting Data into Apache Iceberg with Upsolver

AWS Glue

AWS Glue is a fully managed ETL service that simplifies data ingestion by automatically discovering, cataloging, and transforming data from various sources for seamless integration into your data lake or data warehouse.

Articles About Ingesting Data into Iceberg with AWS Glue:

• Ingesting Data into Apache Iceberg with AWS Glue
• Ingesting Streaming Data from AWS Kinesis into Apache Iceberg with AWS Glue
• BI Dashboards with AWS Glue and Dremio

Airbyte

Airbyte is an open-source data integration platform that enables easy data ingestion by connecting various data sources and destinations with customizable, pre-built connectors, facilitating efficient and scalable data pipelines.

Articles About Ingesting Data into Iceberg with Airbyte:

• Ingesting Data into Apache Iceberg with Airbyte OSS

Fivetran

Fivetran is a fully managed data integration service that automates data ingestion by continuously syncing data from various sources into your data warehouse or lakehouse, ensuring reliable and up-to-date data pipelines.

Articles About Ingesting Data into Iceberg with FiveTran:

• Using Fivetran with Apache Iceberg
• GDW: Using Fivetran with Apache Iceberg

Conclusion

Apache Iceberg has an expansive ecosystem, and this article provides an overview of eight powerful tools that can facilitate data ingestion into Apache Iceberg and offers resources to help you get started. Whether leveraging Dremio's comprehensive lakehouse platform, using open-source solutions like Apache Spark or Kafka Connect, or integrating with managed services like Upsolver and Fivetran, these tools offer the flexibility and scalability needed to build and maintain an efficient and effective data lakehouse environment.

Contact us today to do a free architectural workshop and discover what tools will meet your needs.