Shulou(Shulou.com)06/02 Report--

This article mainly explains "what are the functions of the distributed task scheduling platform XXL-JOB". The content of the explanation in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's ideas to study and learn what are the functions of the distributed task scheduling platform XXL-JOB.

Basic introduction

In project development, the following scenarios often require distributed task scheduling:

When the tasks of multiple instances of the same service are mutually exclusive, they need to be unified and coordinated.

The execution of scheduled tasks needs to support high availability, monitoring operation and maintenance, and fault alarm.

It is necessary to uniformly manage and track the operation of the timing tasks of each service node, as well as the task attribute information, such as the service to which the task belongs and the person responsible.

Therefore, XXL-JOB arises at the historic moment: XXL-JOB is an open source lightweight distributed task scheduling platform, its core design goal is rapid development, easy to learn, lightweight, easy to expand, easy to use out of the box, in which "XXL" is the main author, the abbreviation of Dianping and Xu Xueli's name.

Since the open source in 2015, it has been connected to the online product lines of hundreds of companies, and the access scenarios involve e-commerce business, O2O business and big data homework.

Functional characteristics

The main functional features are as follows:

Simple and flexible Web pages are provided to manage tasks. The management system supports user management and permission control, supports container deployment, and supports cross-platform task scheduling through general HTTP.

Rich task management functions support page-facing task CRUD operations; support scripting tasks, command line tasks, Java code tasks and execution on the page; support task cascading, trigger subtask execution after parent task execution is over; support setting task priority; support setting specific task execution node routing policies, including polling, random, broadcast, failover, busy transfer, etc. Support Cron mode, task dependency, scheduling center API interface mode to trigger task execution

The high-performance scheduling center triggers scheduling tasks based on thread pool multi-thread; fast tasks and slow tasks are scheduled separately based on thread pool to provide system performance and stability; fully asynchronous design and implementation of task scheduling process, such as asynchronous scheduling, asynchronous operation, asynchronous callback, etc., effectively peak traffic for dense scheduling.

Highly available task scheduling centers and task execution nodes are all deployed in clusters, supporting dynamic expansion and failover, supporting task configuration routing and failover strategies, the unavailability of actuator nodes is automatically transferred to other nodes to support task timeout control, failure retry configuration supports task processing blocking strategy: schedule processing strategies that are too late to execute tasks when the task execution node is busy Including: serial, discard, override strategy

Easy to monitor, operation and maintenance support setting task failure email alarm, reserved interface to support SMS and nail alarm, real-time view of task execution operation data statistics chart, task progress monitoring data, task complete execution log

Design idea of system Design 1

The scheduling behavior is abstracted into a common platform of "scheduling center", but the platform itself does not undertake business logic, and the "scheduling center" is responsible for initiating scheduling requests; the tasks are abstracted into decentralized JobHandler, which is uniformly managed by the "executor", and the "executor" is responsible for receiving scheduling requests and executing the business logic in the corresponding JobHandler. Therefore, the "scheduling" and "task" parts can be decoupled from each other to improve the overall stability and expansibility of the system.

2 system composition

Scheduling module (scheduling center): responsible for managing scheduling information, issuing scheduling requests according to scheduling configuration, and does not bear the business code itself. The scheduling system is decoupled from the task, which improves the availability and stability of the system, and the performance of the scheduling system is no longer limited by the task module. It supports visual, simple and dynamic management and scheduling information, including task creation, update, deletion, task alarm, etc., all of which will take effect in real time. At the same time, it supports monitoring scheduling results and execution logs, and supports executor Failover.

Execution module (executor): responsible for receiving scheduling requests and executing task logic. The task module focuses on operations such as task execution, which makes development and maintenance more simple and efficient, and receives execution requests, termination requests and log requests from the "scheduling center".

3 working principle

The task executor automatically registers with the scheduling center according to the address of the configured scheduling center.

When the task trigger condition is reached, the dispatch center sends the task.

The executor executes tasks based on the thread pool, puts the execution results in the memory queue, and writes the execution log to the log file.

The callback thread of the executor consumes the execution result in the memory queue and actively reports it to the scheduling center.

When the user views the task log in the scheduling center, the scheduling center requests the task executor, which reads the task log file and returns the log details.

4 HA Design 4.1 Dispatch Center High availability

The scheduling center supports multi-node deployment and ensures that only one scheduling center node triggers task scheduling at the same time based on the database row lock. See com.xxl.job.admin.core.thread.JobScheduleHelper#start

Connection conn = XxlJobAdminConfig.getAdminConfig (). GetDataSource (). GetConnection (); connAutoCommit = conn.getAutoCommit (); conn.setAutoCommit (false); preparedStatement = conn.prepareStatement ("select * from xxl_job_lock where lock_name = 'schedule_lock' for update"); preparedStatement.execute (); # trigger task scheduling # transaction submission conn.commit (); 4.2 task scheduling highly available

The routing policy scheduling center selects an executor node to execute tasks based on routing policies. XXL-JOB provides the following routing policies to ensure high availability of task scheduling: busy transfer strategy: initiating a rpc heartbeat request to the executor node before sending a task to query whether it is busy If the actuator node returns busy, transfer to another actuator node to implement the failover strategy (see com.xxl.job.admin.core.route.strategy.ExecutorRouteBusyover): before issuing the task, send a rpc heartbeat request to the actuator node to check whether it is online, and if the actuator node does not return or is not available, transfer to another actuator node for execution (refer to com.xxl.job.admin.core.route.strategy.ExecutorRouteFailover)

Blocking policy when the executor node has multiple tasks with the same id that have not been completed, the task needs to be chosen based on the blocking policy: serial policy: default policy, task queuing, discarding old task policy, discarding new task policy (see: com.xxl.job.core.biz.impl.ExecutorBizImpl#run)

Similar framework comparison features quartzelastic-job-litexxl-jobLTS relies on MySQL, jdkjdk, zookeepermysql, jdkjdk, zookeeper, maven highly available multi-node deployment, through competitive database locks to ensure that only one node performs tasks through zookeeper registration and discovery, servers can be dynamically added based on competitive database locks to ensure that only one node executes tasks, supporting horizontal expansion. You can manually add scheduled tasks, start and pause tasks, monitor cluster deployment, and dynamically add servers. You can manually add scheduled tasks, start and pause tasks. Monitoring task fragmentation × √√√ management interface × √√√ is simple and slightly complex advanced functions-elastic expansion, multiple job modes, failure transfer, running state collection, multithreaded data processing, idempotency, fault tolerance, spring namespace support elastic expansion, sharding broadcast, failover, Rolling real-time log, GLUE (support for online code editing, no release), task progress monitoring Task dependence, data encryption, email alarm, running reports, internationalization support spring,spring boot, business logger, SPI extension support, failover, node monitoring, diversified task execution result support, FailStore fault tolerance, dynamic expansion. Version update not updated for half a year, no update for 2 years, no update recently, no update for 1 year, use quick start.

How to get started quickly? official document: http://www.xuxueli.com/xxl-job/ has been introduced in more detail and clearly, so I won't repeat it.

Matters needing attention

1 clock synchronization problem scheduling center and task executor need time synchronization, and the synchronization time error needs to be within 3 minutes, otherwise an exception is thrown. Reference: com.xxl.rpc.remoting.provider.XxlRpcProviderFactory#invokeService

If (System.currentTimeMillis ()-xxlRpcRequest.getCreateMillisTime () > 3 / 60 / 1000) {xxlRpcResponse.setErrorMsg ("The timestamp difference between admin and executor exceeds the limit."); return xxlRpcResponse;}

The 2 time zone problem task is triggered by the scheduling center. When triggered according to the cron expression of the task set in the scheduling center, you need to pay attention to the time zone where the machine deploying the scheduling center is located, and customize the cron expression according to this time zone

(3) the problem of service downtime in task execution is that the scheduling center completes the task dispatch, and if the executor suddenly fails in the process of executing the task, it will lead to the execution problem of the task in the scheduling center. The scheduling center does not initiate a failed retry. Even if the task sets the timeout, and the actuator fails to cause the task to be unfinished for a long time, the scheduling center interface will not see the task timeout, because the task timeout is detected by the executor and reported to the scheduling center.

Therefore, when you encounter a task that has not been completed for a long time, you can pay attention to whether the actuator suddenly fails to serve.

4 graceful downtime problem the executor executes tasks asynchronously based on the thread pool. When you need to restart, you need to pay attention to the problem of unfinished tasks in the thread pool. You need to stop gracefully, and you can stop gracefully directly based on XxlJobExecutor.destroy (). Note that in the version before v2.0.2, there is bug that can't stop gracefully. Fix v2.0.2 and later (reference: https://github.com/xuxueli/xxl-job/issues/727)

5 failure retry problem when some services of the actuator node are not available, such as the node disk is damaged, but when the scheduling center is still online, the scheduling center may still be routed to the unoffline node based on routing policies (including failover policies). And keep retrying, constantly failing, resulting in exhaustion of the number of retries. Therefore, the routing strategy should not be fixed, such as the first one and the last one.

Thank you for your reading, the above is the content of "what are the functions of the distributed task scheduling platform XXL-JOB". After the study of this article, I believe you have a deeper understanding of the functions of the distributed task scheduling platform XXL-JOB, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.