Shulou(Shulou.com)06/03 Report--

Easy Scheduler, a distributed workflow task scheduling system, is officially open source

1. Background

With the efforts of many technology partners, after nearly 2 years of research and development iteration, internal business divestiture and refactoring, and after a trial period by a group of seed users, EasyScheduler finally ushered in its first official open source release-1.0.0.

It is believed that partners who have done data processing should be familiar with open source scheduling systems such as oozie, azkaban and airflow. In using these scheduling systems, they may have such experiences: for example, the configuration workflow task cannot be visualized, the running status of the task cannot be viewed online in real time,

The task cannot be paused while running, cannot support parameter passing, cannot be replenished, cannot be used by multi-tenants, and the scheduling system is not highly available, and so on. It is under this background that Easy Scheduler came into being, and its goal is to make scheduling more easy, and we can see our original intention from its Chinese name "easy scheduling".

2. Design features

Easy Scheduler is a distributed workflow task scheduling system, which mainly solves all kinds of problems caused by the complex dependency relationship of data research and development ETL.

Its main objectives are as follows:

By correlating Task according to task dependency in the way of DAG diagram, you can visually monitor the running status of tasks in real time and support rich task types: Shell, MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python,Sub_Process, Procedure, etc. support workflow timing scheduling, dependent scheduling, manual scheduling, manual pause / stop / resume. At the same time, support for failure retry / alarm, recovery failure from specified nodes, Kill tasks and other operations to support workflow priority, task priority and task failover, and task timeout alarm / failure support workflow global parameters and node custom parameter settings support online upload / download of resource files, management, etc. Support for online file creation, editing, online viewing and scrolling of task logs, online downloading of logs and so on to achieve cluster HA, Master cluster and Worker cluster decentralization support for Master/Worker cpu load,memory through Zookeeper Cpu online view support workflow running history tree / Gantt chart display, support task status statistics, process status statistics support complement support multi-tenant support internationalization and more partners to explore 4, comparison with similar scheduling systems

5. System partial screenshot

6. Easy Scheduler back-end deployment document Easy Scheduler front-end deployment document Easy Scheduler user manual

For more documentation, please refer to:

Easy Scheduler Chinese online documents

7. Thank you

Easy Scheduler uses many excellent open source projects, such as google's guava, guice, grpc,netty,ali 's bonecp,quartz, and apache's many open source projects. We are also very grateful to oozie, azkaban, airflow and other excellent scheduling works for their inspiration.

It is because of standing on the shoulders of these open source projects that the birth of Easy Scheduler is possible. We are very grateful to all the open source software used for this! We also hope that we can not only benefit from open source, but also become open source.

Contributor, so we decided to contribute to easy scheduling and committed to long-term maintenance. I also hope that partners who have the same enthusiasm and belief in open source will join us to contribute to open source!

8. Postscript

Just a week after Easy Scheduler officially opened up on March 28, 2019.028, we felt the great enthusiasm of our partners for Easy Scheduler. Many partners proposed to use feedback, and some partners directly found the corresponding source code to ask questions or give better suggestions, or even rolled up their sleeves to write code on Easy Scheduler.

This is a great spiritual encouragement to our current major developers. Thank you very much for your enthusiasm and trust in us. We will continue to work with you on the road of making the dispatching system available out of the box. To make "data capabilities civilian", to contribute their passion and sweat to the data era!

Open source github address: https://github.com/analysys/EasyScheduler

Online document address: https://analysys.github.io/easyscheduler_docs_cn

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.