Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to solve the distributed task scheduling xxl-job problem". The content is simple and easy to understand, and the organization is clear. I hope it can help you solve your doubts. Let Xiaobian lead you to study and learn this article "how to solve the distributed task scheduling xxl-job problem".

xxl-job profile

XXL-JOB is a lightweight distributed task scheduling framework. Its core design goals are rapid development, simple learning, lightweight, and easy to expand. Now open source and access to a number of companies online product line, out-of-the-box.

functions as follows

1, simple: support CRUD operation of tasks through Web pages, easy to operate, one minute to get started;

2. Dynamic: support dynamic modification of task status, suspension/resumption of tasks, and termination of running tasks, effective immediately;

3. Dispatching center HA (central type): dispatching adopts central design, and "dispatching center" is realized based on cluster Quartz, which can ensure dispatching center HA;

4. Executor HA (distributed): task distributed execution, task "executor" supports cluster deployment, which can ensure task execution HA;

5. Task Fail: When the executor cluster is deployed, the task routing policy selects "Fail over". When the scheduling fails, the executor will switch smoothly for Fail.

6. Consistency: "Scheduling Center" ensures consistency of cluster distributed scheduling through DB lock, and one task scheduling will only trigger one execution;

7. User-defined task parameters: support online configuration scheduling task input parameters, effective immediately;

8, scheduling thread pool: scheduling system multi-thread trigger scheduling operation, to ensure accurate scheduling execution, not blocked;

9. Flexible expansion and contraction: Once a new actuator machine is online or offline, the task will be reassigned at the next scheduling;

10. Email alarm: Support email alarm when task fails, support configuring multi-email address group alarm email;

11. Status monitoring: support real-time monitoring of task progress;

12. Rolling execution log: Support online viewing of scheduling results, and support real-time viewing of the complete execution log of actuator output in Rolling mode;

13. GLUE: Provide Web IDE, support online development of task logic code, dynamic release, real-time compilation into effect, omit the process of deployment online. Support 30 versions of historical version backtracking.

14. Data encryption: The communication between the dispatching center and the actuator is encrypted to improve the security of dispatching information;

15. Task Dependency: Support configuring subtask dependency. When the parent task is completed and successfully executed, it will actively trigger the execution of a subtask. Multiple subtasks are separated by commas.

16, push maven central repository: will push the latest stable version to the maven central repository, convenient for users to access and use;

17, task registration: the executor will automatically register the task periodically, the dispatch center will automatically discover the registered task and trigger the execution. At the same time, it also supports manual entry of actuator address;

18. Routing policy: Provide rich routing policies when implementing cluster deployment, including: first, last, polling, random, consistent HASH, least frequently used, most recently unused, failover, busy transfer, etc.;

19. Operation report: Support real-time viewing of operation data, such as task quantity, scheduling times, executor quantity, etc.; and scheduling report, such as scheduling date distribution chart, scheduling success distribution chart, etc.;

20. Script tasks: Support developing and running script tasks in GLUE mode, including Shell, Python and other types of scripts;

21. Blocking processing strategy: The processing strategy when the scheduling is too dense and the executor has no time to process. The strategies include: single serial (default), discarding subsequent scheduling, and overriding the previous scheduling;

22. Failure handling strategy: handling strategy when scheduling fails, including failure alarm (default) and failure retry;

23. Fragmented Broadcast Task: When the executor cluster is deployed, if the task routing strategy selects "Fragmented Broadcast," a task scheduling will broadcast and trigger all executors in the cluster to execute a task once. Fragmented tasks can be developed according to fragmentation parameters.

24. Dynamic fragmentation: Fragmented broadcast tasks are fragmented with executors as the dimension, and support dynamic expansion of executor clusters to dynamically increase the number of fragments for collaborative business processing; it can significantly improve task processing capacity and speed when performing large data volume business operations.

25. Event triggering: In addition to "Cron mode" and "task dependent mode" to trigger task execution, event-based task triggering mode is supported. The dispatch center provides API services that trigger task execution once, which can be triggered flexibly according to business events.

quick start

Project address: github.com/xuxueli/xxl-job

2.1 Initialize Dispatch Database

Please download the project source code and extract it, obtain the "Scheduling Database Initialization SQL Script" and execute it. Under normal circumstances, 16 tables should be generated.

"Scheduling Database Initialization SQL Script" is located at:

/xxl-job/doc/db/tables_xxl_job.sql

The dispatch center supports cluster deployment. In case of cluster, each node must be connected to the same mysql instance.

If mysql is the master slave, the dispatching center cluster node must be forced to go to the master library;

2.2 compile source code

Extract the source code, import the source code into IDE according to maven format, and compile it using maven. The source code structure is as follows:

xxl-job-admin: dispatch center

xxl-job-core: public dependency

xxl-job-executor: Executor Sample (Select the appropriate version of the actuator, you can use it directly, or you can refer to it and transform the existing project into an actuator)

: xxl-job-executor-sample-spring: Spring version, through Spring container management executor, more common, recommended this way;

: xxl-job-executor-sample-springboot: Springboot version, executor management via Springboot;

: xxl-job-executor-sample-jfinal: JFinal version, executor management via JFinal;

2.3 Configure Deployment Dispatch Center

Dispatching center project: xxl-job-admin

Role: Unify task scheduling on task scheduling platform, trigger scheduling execution, and provide task management platform.

Step 1: Dispatching Center Configuration:

Dispatch Center Configuration File Address:

/xxl-job/xxl-job-admin/src/main/resources/xxl-job-admin.properties

Dispatching Center Configuration Description:

###Dispatch Center JDBC Link: Please keep the link address consistent with the address of the dispatch database created in Section 2.1

xxl.job.db.driverClass=com.mysql.jdbc.Driver

xxl.job.db.url=jdbc:mysql://localhost:3306/xxl-job? useUnicode=true&characterEncoding=UTF-8

xxl.job.db.user=root

xxl.job.db.password=root_pwd

###Alarm mailbox

xxl.job.mail.host=smtp.163.com

xxl.job.mail.port=25

xxl.job.mail.username=ovono802302@163.com

xxl.job.mail.password=asdfzxcv

xxl.job.mail.sendFrom=ovono802302@163.com

xxl.job.mail.sendNick= Task Scheduling Platform XXL-JOB

###Login Account

xxl.job.login.username=admin

xxl.job.login.password=123456

###Dispatch center communication TOKEN, enabled when not empty

xxl.job.accessToken=

Step 2: Deploy the project:

If you have configured this correctly, you can compile the project into a war package and deploy it to tomcat. Dispatching center access address: http://localhost:8080/xxl-job-admin (this address will be used by the executor as the callback address), the operation interface after logging in is shown in the following figure.

So far, the "dispatch center" project has been successfully deployed.

Step 3: Dispatch Center Cluster (optional):

The dispatch center supports cluster deployment and improves the availability of the dispatch system.

The only requirement for cluster deployment is to ensure that the configuration of each cluster node (db and login account, etc.) is consistent. The dispatch center distinguishes different clusters by db configuration.

The dispatch center can use nginx Load Balancer during cluster deployment. In this case, a domain name can be assigned to the cluster. This domain name can be used for access on the one hand and for configuring executor callback addresses on the other.

2.4 Configure Deployment Actuator Project

"Executor" project: xxl-job-executor-sample-spring (if you create a new actuator project, please refer to the configuration steps of the Sample actuator project;)

Role: Responsible for receiving and executing the dispatch from the dispatch center; can directly deploy the executor, or integrate the executor into the existing business project.

Step 1: maven dependency confirmation pom file introduced "xxl-job-core" maven dependency;

Step 2: actuator configuration actuator configuration, configuration file address:

/xxl-job/xxl-job-executor-samples/xxl-job-executor-sample-spring/src/main/resources/xxl-job-executor.properties

Actuator configuration, configuration description:

### xxl-job admin address list: Dispatching center deployment and address: if there are multiple addresses in dispatching center cluster deployment, they shall be separated by commas. The Executor will use this address for Executor Heartbeat Registration and Task Result Callback.

xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin

### xxl-job executor address: executor "AppName" and address information Configuration: AppName Executor heartbeat registration grouping basis; address information is used for "dispatch center request and trigger task" and "executor registration." The default port of the actuator is 9999, and the default IP of the actuator is blank, which means that the IP is automatically obtained. If there are multiple network cards, the specified IP can be manually set, and the Host will be bound when the IP is manually set. When deploying multiple actuators on a single machine, note that different actuator ports should be configured;

xxl.job.executor.appname=xxl-job-executor-sample

xxl.job.executor.ip=

xxl.job.executor.port=9999

### xxl-job log path: The disk location where the executor log file is stored. You need to have read and write permissions on this path.

xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler/

### xxl-job, access token: actuator communication TOKEN, enabled when not empty

xxl.job.accessToken=

Step 3: Configure the actuator component. Configuration file address:

/xxl-job/xxl-job-executor-samples/xxl-job-executor-sample-spring/src/main/resources/applicationcontext-xxl-job.xml

Actuator component, configuration description:

Step 4: Deploy the executor project: If the above configuration has been correctly performed, the executor project can be compiled and deployed. The system provides three executor sample projects, and one of them can be selected. The respective deployment methods are as follows.

xxl-job-executor-sample-spring: The project is compiled and packaged into a WAR package and deployed into tomcat.

xxl-job-executor-sample-springboot: the project is compiled and packaged into an executable JAR package of springboot type, and the command can be started;

At this point, the deployment of the "actuator" project has been completed.

Step 5: Actuator Cluster (optional):

Executors support cluster deployment, improve scheduling system availability, and improve task processing capacity.

The only requirement for cluster deployment is to ensure that the configuration item "xxl.job.admin.addresses/dispatch center address" of each actuator in the cluster is consistent, and the actuators perform operations such as automatic actuator registration according to this configuration.

2.5 Develop the first mission "Hello World"

This example takes the task of creating a new GLUE mode (Java) run mode. For more details about the mission configuration, please see Chapter 3: Mission Details. (The execution code of "GLUE mode (Java)" is hosted to the dispatch center for online maintenance, which is simpler and lighter than that of "Bean mode task" which needs to be deployed online in the executor project development)

Premise: Please confirm that the "Dispatch Center" and "Executor" projects have been successfully deployed and started;

Step 1: Create a new task: Log in to the Dispatch Center and click the New Task button shown in the figure below to create a new sample task. Then, refer to the task parameter configuration in the screenshot below and click Save.

Step 2:"GLUE Mode (Java)" Task Development:

Please click the "GLUE" button on the right side of the task to enter the "GLUE Editor Development Interface," as shown in the figure below. By default, tasks in GLUE mode (Java) run mode already initialize the sample task code, which prints Hello World. (The task in "GLUE Mode (Java)" is actually a Java class code inherited from IJobHandler, which runs in the executor project and can be injected into other services in the executor using @Resource/@Autowire)

Step 3: Trigger execution: Click the "Execute" button on the right side of the task to manually trigger the execution of a task (usually, the task is scheduled by configuring the Cron expression).

Step 4: View Log: Click Log on the right side of the task to view the task log. In the task log interface, you can view the historical scheduling records of the task and the task scheduling information, execution parameters and execution information of each scheduling. For running tasks, click the "Execution Log" button on the right to enter the log console to view the real-time execution log.

The above is "how to solve the distributed task scheduling xxl-job problem" all the content of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.