Shulou(Shulou.com)06/03 Report--

Background

Whether Internet applications or enterprise applications, there are a large number of batch processing tasks. Some task scheduling systems are often needed to help developers solve problems. With the gradual evolution of micro-service architecture, single architecture has gradually evolved into distributed and micro-service architecture. In this context, many of the original task scheduling platforms have been unable to meet the needs of business systems. So there are some distributed task scheduling platforms. Each of these platforms has its own characteristics, but each has its own shortcomings, such as not supporting task scheduling, high coupling with business, not supporting cross-platform and so on. It does not meet the needs of the new generation of micro-service architecture, so Yixin developed a micro-service task scheduling platform (SIA-TASK).

SIA is the abbreviation of Simple is Awesome, the basic development platform of Yixin Company, and SIA-TASK (micro-service task scheduling platform) is one of the important products. SIA-TASK accords with the current micro-service architecture model, and has the characteristics of cross-platform, orchestration, high availability, non-intrusion, consistency, asynchronous parallel, dynamic expansion, real-time monitoring and so on.

Project profile

SIA-TASK is an integrated solution for task scheduling. Carry on the metadata collection to the task, then carry on the task visual arrangement, finally carry on the task scheduling, and take the whole process monitoring to the task, it is easy to use. There is no intrusion into the business, and a task scheduling model in line with expectations can be generated through simple and flexible configuration.

Drawing lessons from the design idea of micro-service, SIA-TASK obtains the task metadata distributed on each task executor and uploads it to the task registry. Using online task scheduling, we can dynamically modify the task clock, use HTTP as the task scheduling protocol, uniformly use JSON data format, and the scheduling center parses the clock, executes the task flow, and notifies the task.

Key term task (Task): basic execution unit, an HTTP call interface exposed by the executor; Job: consisting of one or more tasks with logical relationships (serial / parallel), the smallest unit scheduled by the task scheduling center; Plan: composed of several sequentially executed jobs, each job has its own execution cycle, and the plan has no execution cycle Task scheduling center (Scheduler): scheduling according to the execution cycle of each job, that is, making HTTP requests according to the logic of plans, jobs, and tasks; task scheduling center (Config): task scheduling center uses tasks to create plans and jobs; task executor (Executer): receives HTTP requests for business logic execution The Hunter:Spring project expansion package is responsible for capturing tasks in the executor and uploading to the registry. The business can rely on this component for Task writing. The microservice task scheduling platform automatically crawls tasks based on annotations. @ OnlineTask annotations are added to the methods exposed as HTTP services. @ OnlineTask automatically grabs the IP address, port, request path, request method, request parameter format and other information of the method to the task registry (zookeeper), and writes it to the persistent storage synchronously. This method is the task. Based on annotated non-intrusive multithreaded control, a single task instance must be run in a single thread, and the task scheduling framework automatically intercepts @ OnlineTask annotations for single-thread running control, so that a task will not be scheduled again when it is running. And the whole control process is completely unaware of the developer. Scheduler adaptive task assignment, failure during task execution, exception. The task can be reawakened at multiple points according to the strategy customized by the task to ensure the uninterrupted execution of the task. Highly flexible task scheduling mode, the design idea of SIA-TASK is to take tasks as atoms and combine multiple tasks according to the relationship of execution to form a job. At the same time, the runtime is divided into task scheduling center and task scheduling center, so that job scheduling and job scheduling are separated and do not affect each other. When we need to adjust the flow of the job, we only need to process it in the choreography center. At the same time, the orchestration center supports the organization of tasks according to serial, parallel, branch and other ways. Multiple scheduling methods are also supported for different task instances of the same task. Design of micro-service task scheduling platform

SIA-TASK is mainly divided into five parts:

Task executor task scheduling center task scheduling center task registry (zookeeper) persistent storage (Mysql)

The main running logic of SIA-TASK:

By annotating the tasks in the task executor, report to the task registry task orchestration center to obtain data from the task registry center and save it to the persistent storage task scheduling center to obtain scheduling information from the persistent storage task scheduling center to access the task executor according to the scheduling logic.

UI Preview

The home page provides multi-dimensional monitoring.

Scheduler information: displays scheduler information (load capacity, early warning value), as well as job distribution. Scheduling information: displays the scheduling times triggered by the scheduling center and the multi-dimensional scheduling statistics of jobs and tasks. Docking project statistics: count the systems that use the project, the number of jobs, the number of tasks, and so on.

Scheduling monitoring provides real-time monitoring and display of submitted jobs.

Real-time monitoring of job status: take the project team as a unit panel to display the running status of the job. Real-time log association: the real-time log association can be displayed through the coloring status icon.

Task management: provide operations related to task metadata

Task metadata entry: tasks in manual mode, which can be entered here. Task connectivity test: provide task connectivity test. Other operations of task metadata: modify, delete.

Job management: provide job-related operations

Task scheduling: the arrangement of assignments. Publish jobs: create, modify, and publish jobs. Cascading settings: provide job settings that are time dependent.

Log management

Local log: the log interface is simple and the query is fast; the log provides a 7-day scheduling log for quick query. Open source address https://github.com/siaorg/sia-task

Author: Liang Xin, head of Yixin Development platform / Senior architect

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.