Shulou(Shulou.com)06/01 Report--

This article focuses on "how to introduce a security checkpoint mechanism for kill job". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to introduce a security checkpoint mechanism for kill job.

About XXL-JOB-ONION

XXL-JOB-ONION is based on the secondary development of XXL-JOB, we have made extensions based on the secondary development of XXL-JOB, such as adding ONION_BEAN running mode and improving the alarm function.

About the ONION_BEAN mode:

For scheduled tasks that do not require sharding: implemented by inheriting the OnionJobHandler interface instead of using the annotations provided by XXL-JOB

For scheduled tasks that require sharding: implemented by inheriting the OnionShardingJobHandler interface instead of using the annotations provided by XXL-JOB

Background

According to the investigation of scheduled tasks in our current project, we find that most scheduled tasks have one thing in common: each scheduled task is processed by querying a batch of data from the database, and each piece of data is converted into a Runable to be executed in a business-isolated thread pool, for example:

@ Component

Public class OnionXxlJob implements OnionJobHandler {

@ Override

Public void doExecute (JobParam param) throws Exception {

/ / actually query from the database

List orderIds = Arrays.asList (10000, 11111)

/ / parallel flow simulation is used here to put each order into a thread pool for processing.

OrderIds.parallelStream () forEach (orderService::handlerOrder)

}

}

Since scheduled tasks need to deal with large amounts of data each time, which may take an hour and a half to execute, to solve the embarrassment of waiting for scheduled task execution to be completed and afraid of kill for kill when you need to modify the configuration and restart each version update or in response to a sudden amount of data, we introduce a security checkpoint mechanism for xxl-job, which is inspired by the security checkpoint mechanism of jvm garbage collection.

Why do you think kill dare not kill?

Although all operations involving the database need to be placed in the transaction, not all operations can be rolled back. For example, operations that modify redis cached data and ES data cannot be rolled back (usually those that can be put into the transaction commit are later executed). If you need to call a third-party interface in the process of data processing, it will be more complicated (we have already achieved this through the playback mechanism).

Suppose hundreds of threads are working at the same time, and suddenly kill drops the task in progress, which means that there may be problems with hundreds of data processing, and we have to find the data and complete the unfinished steps.

Outline design

Let's first analyze the kill job process of XXL-JOB: the xxl-job-admin front end initiates a kill request, and after receiving it, the admin sends a kill request to the executor and carries the jobId. The executor determines whether the current task is executed again according to the jobId, and cancels if it is.

Is there a solution that when a scheduled task administrator initiates a kill job operation from the background, wait for the task to enter a safe point before kill the task?

Let's take a look at the following picture:

Process description:

1. The administrator initiates the kill request to the executor during the admin operation, or the executor itself initiates the kill request to itself to implement the blocking policy.

2. After receiving the command, the executor first obtains the JobHandler and notifies the security checkpoint administrator of the JobHandler to prevent the new Runable from entering the checkpoint

3. Ask the security checkpoint counter of JobHandler to realize that the value of the blocking wait counter is 0 and then kill, or the asynchronous wait counter is 0 and then kill

4. XXL-JOB assigns an ID to each Job, and each JobId holds a counter to isolate tasks triggered in different time periods from the Job. Of course, XXL-JOB does not allow multiple identical Job to be executed at the same time.

On the design of security checkpoints

Security checkpoints: business methods invoked by Runable in scheduled tasks

For example, if you regularly close the scheduled task of overdue orders, a method to close the order will be called for each time-out order queried, so this method can be set as a security checkpoint.

Because the security checkpoint needs to be combined with business considerations, we only provide a checkpoint annotation in our design, so that users can decide which method to set as a security checkpoint and whether they need to set a security checkpoint.

Cons: through AOP, only the method of public can be set as a security checkpoint.

Consider using bytecode enhancement to force the check behavior into the private method, but it may cause the application to start slower, and we'll consider this approach if necessary.

However, we also provide programming to solve this shortcoming, and like the transaction mechanism implemented by Spring, security checkpoints support the use of annotations as well as programmatic use.

About the design of the security check administrator role:

Only counting the current number of threads entering the security checkpoint method through the counter can not achieve "atomicity". In the process of task execution, a new Runable enters the security checkpoint method every moment, and Runable comes out of the security checkpoint method all the time.

The security check administrator controls whether Runable can enter the security checkpoint method, and does not let Runable enter the security checkpoint before waiting for the counter value to change to 0, which can solve the problem of "atomicity" waiting.

Two ways of use

1. Use of annotations

The use of annotations requires that the method must be public, which, like Spring annotated transactions, is not valid.

2. Programmatic use

Programmatic use also requires only three steps:

1. First determine whether there is a security checkpoint flag, and if so, you can abandon the execution.

2. Add one to the counter, which is the default method defined in the OnionShardingJobHandler or OnionJobHandler interface

3. The counter minus one, which is the default method defined in the OnionShardingJobHandler or OnionJobHandler interface

At this point, I believe you have a deeper understanding of "how to introduce a security checkpoint mechanism for kill job". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.