Apache DolphinScheduler

What is Apache DolphinScheduler?

Apache DolphinScheduler is an open-source distributed job scheduling system developed by the Apache Software Foundation. It provides a dedicated solution for solving complex scheduling dependencies in the data pipeline, across several big data business scenarios.

History

Apache DolphinScheduler (initially named EasyScheduler) was started in 2018 by Analysys. It was taken over by the Apache Software Foundation in 2019, making it a powerful, community-driven project. Over time, it has seen multiple version updates, refining its capabilities, and enhancing its user interface for optimal user interaction.

Functionality and Features

Apache DolphinScheduler provides a host of features that are tailored for big data processing:

  • Supports various types of tasks like shell, MR, spark, SQL (MySQL, postgres, hive, spark SQL), python, sub_process.
  • Depicts a visuals DAG that shows task dependencies.
  • Offers a drag-and-drop interface for creating task workflows.
  • Supports multi-tenant system architecture.
  • Provides a global timing scheduling and cron-like scheduling.

Architecture

DolphinScheduler follows a decentralized multi-master and multi-worker system architecture. This structure allows distributed scheduling, along with efficient resource management and task scheduling. It also includes alert server components and an API module to expose its functionalities to other systems.

Benefits and Use Cases

DolphinScheduler is widely used for Big Data and data-driven scenarios due to its high availability, horizontal scalability, and support for various task types. Use cases include time series data processing, ETL processes, machine learning workflows, and data backup processes.

Challenges and Limitations

While powerful, DolphinScheduler faces challenges around the difficulty of debugging workflows and a limited community, with fewer resources for learning and troubleshooting than other established Apache projects.

Integration with Data Lakehouse

In a Data Lakehouse environment, DolphinScheduler provides efficient orchestration and scheduling of data jobs. These jobs could range from simple ETL tasks to complex big data processing and analytics tasks.

Security Aspects

DolphinScheduler provides role-based access control for different modules, such as processes, resources, and data sources, ensuring secure operations across multi-tenant environment.

Performance

With its decentralized design, DolphinScheduler ensures robust performance, even in large-scale data processing scenarios, by distributing the workload across multiple nodes.

FAQs

What kind of tasks does Apache DolphinScheduler support? DolphinScheduler supports a wide range of tasks including shell, MR, Spark, SQL (MySQL, Postgres, Hive, Spark SQL), Python, and Sub-process.

How does Apache DolphinScheduler integrate with a data lakehouse? It orchestrates and schedules data jobs efficiently, ranging from simple ETL tasks to complex big data processing and analytics tasks.

What are the security measures in Apache DolphinScheduler? It provides role-based access control for various modules ensuring secure operations across multi-tenant environment.

Glossary

Decentralized Multi-master and Multi-worker System: A system architecture that enables the distribution of tasks across multiple nodes for efficient scheduling and resource management.

Data Lakehouse: A blend of data warehouse and data lake technologies, providing structured and semi-structured data processing.

DAG: Directed Acyclic Graph, used in DolphinScheduler to represent task dependencies.

ETL: Extract-Transform-Load, a type of data integration strategy.

Role-based Access Control: An approach to restricting system access to authorized users.

get started

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now
demo on demand

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo
talk expert

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us

Ready to Get Started?

Bring your users closer to the data with organization-wide self-service analytics and lakehouse flexibility, scalability, and performance at a fraction of the cost. Run Dremio anywhere with self-managed software or Dremio Cloud.