Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to use K8s for job scheduling, many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

Recently, in the company's data synchronization project (hereinafter referred to as ZDTP), the distributed scheduling data synchronization execution unit needs to be used. at present, the scheme used is to package the data synchronization execution unit into a mirror and use K8s for scheduling.

In ZDTP, the action of data synchronization can be abstracted into an execution unit (hereinafter referred to as worker), which is similar to thread execution unit Runnable. Runnable is put into a queue to wait for the scheduled execution of threads, and its mission is completed after the execution of Runnable. When a user creates a synchronization task in the ZDTP console and starts the task, several worker for the synchronization task will be generated according to the configuration of the synchronization task. Assuming that these worker are executed locally, you can package them into a Runnable, and then create a thread for execution, as shown in the following figure:

However, in stand-alone mode, performance bottlenecks are encountered, and distributed scheduling is required to schedule worker to other machines for execution:

The question is how can we better schedule worker to other machines for execution?

Investigation of Worker deployment method

1. Deploy Worker based on virtual machine

Worker runs in a virtual machine created in advance. When the task starts, you need to select the free Worker according to the current Worker load, which is equivalent to Worker running in the form of Agent, as shown in the following figure:

The following are the main concomitant shortcomings:

The number of Worker Agent is relatively fixed, the cost of creating virtual machines is high, and it is troublesome to expand / reduce capacity. The operation of tasks depends on zk snooping mechanism. If a task fails during operation, you need to implement the failover and automatic restart mechanism to increase the development cycle. Worker Agent load acquisition logic requires project implementation, so it is difficult to obtain load information accurately and increase the development cycle.

2. Deploy Worker based on K8s

Package Worker into a Docker image, schedule jobs to the worker container using K8s, and run only one task per Worker, as shown in the following figure:

The advantages of using K8s are as follows:

The Worker container scheduled by K8s cluster has the function of fault recovery. As long as the restart policy of Pod is set to restartPolicy=Always, K8s will automatically try to restart Worker container no matter what exception occurs during the operation of Worker container, which greatly reduces the cost of operation and maintenance and improves the high availability of data synchronization. Automatically implement the load, for example, when a node has a high load, the Worker container will be scheduled to a node with low load. More importantly, when a node goes down, the workload on it will be automatically transferred to other nodes by K8s. The life cycle of Worker is completely managed by K8s cluster, so you can know the operation of Worker only by calling relevant APIs, which greatly reduces the development cycle. Investigation on scheduling Scheme of K8s Container

The scheduling object of K8s cluster is Pod, and there are many scheduling methods. The following methods are mainly introduced here:

1. Deployment (fully automatic scheduling)

Before talking about Deployment, let's first talk about Replica Set, which is a very important concept of K8s. It is a higher-level abstraction on the abstraction of Pod. Generally, we use the abstraction of Deployment to do the real management of applications, and Pod is the smallest unit that makes up Deployment. It can define a certain Pod (such as Pod wrapped in a ZDTP Worker container) to keep the expected value set by Replica Set at any time. For example, Replica Set can expect to set the number of Pod copies. When the regular inspection of a K8s cluster finds that the number of copies of a certain Pod is less than the expected value set by Replica Set, it will create a Pod instance according to the Pod template set by Replica Set, so that the number of Pod is maintained at the expected value, also through the characteristics of Replica Set. It realizes the high availability of the cluster and reduces the operation and maintenance cost at the same time.

Deployment is implemented internally using Replica Set, which is highly similar to each other, and Deployment can also be thought of as an upgraded version of Replica Set.

2. Job (batch scheduling)

We can define and start a batch task through the k8s Job resource object, process a batch of work items (Work item) in parallel or serially, and the task ends when the processing is complete.

1) Job Template Expansion mode

According to the operation mode of ZDTP Worker, we can use a Job object to correspond to a Worker, and create as many worker as there are. Unless the Pod is abnormal, the Pod will be restarted. After normal execution, the Job will exit, as shown in the figure below:

2) Queue with Pod Per Work Item mode

In this mode, the worker generated by the client is stored in a queue, and then only one job is created to consume the worker item in the queue. By setting the parallelism parameter, how many worker Pod can be started at the same time to process the worker at the same time. What is worth being integrated is that the Worker handler logic in this mode will only pull worker processing from the queue and exit immediately after processing. The completions parameter is used to control the number of Pod normally exited. When the number of Pod exited reaches completions, the Job ends, that is, the completions parameter can control the number of Worker processed by the Job. As shown in the following figure:

3) Queue with Variable Pod Count mode

This scheduling mode looks similar to Queue with Pod Per Work Item mode, but it is not true. As long as a Pod in Queue with Variable Pod Count mode exits normally, it means that Job has finished processing data and is in a terminated state, because each of its Pod has the logic to query whether there is a worker in the queue. Once it is found that there is no worker,Pod exit in the queue, the completions parameter of Queue with Variable Pod Count mode can only be set to 1. The parallelism parameter can start how many worker Pod can process worker at the same time.

This mode also requires that the queue can make Pod aware of whether worker still exists. Message middleware such as RocketMQ/Kafka cannot do this, but will only make the client wait all the time. Therefore, this mode can not choose RocketMQ/Kafka, but can choose database or Redis to implement. As shown in the following figure:

Of course, if there is a need to execute Worker regularly, using cronjob (scheduled task scheduling) of K8s is a very good choice.

3. Pod (default scheduling)

Start the container directly through kind=pod. In this way, you cannot set the number of running instances of the container, that is, replicas = 1. Usually, the production application cluster does not start the container in this way, because the container launched in this way does not have the feature of automatic expansion and scaling of Pod.

It is worth mentioning that even if you have only one copy of Pod, it is officially recommended to use Replica Set for deployment.

Analysis of Pod restart Strategy

Pod's restart strategies include Always, onFailure, and Never:

Always: when the container fails, K8s automatically restarts the container; onFailure: when the container terminates and the exit code is not 0, K8s automatically restarts the container; Never: K8s will not restart the container regardless of its running status

Deployment/Replica Set must be set to Always (because they all need to maintain the number of replicas expected by Pod), while Job can only be set to onFailure and Never to ensure that the container is not restarted after execution. The above three restart policies of the container for direct Pod startup can all be set.

To be clear here, if you use Job, the situation may be a little more complicated:

1) Pod restart policy RestartPolicy=Never

Suppose that Pod exits abnormally during Job scheduling, although the container will not restart at this time. Since Job requires at least one Pod execution to be completed (that is, completions is at least equal to 1), Job is considered complete. Therefore, although an abnormally exited Pod is no longer restarted, Job attempts to restart a Pod execution until the number of normal Pod completions is completions.

$kubectl get pod-namespace zdtp-namespace

NAME READY STATUS RESTARTS AGE

Zdtp-worker-hc6ld 0/1 ContainerCannotRun 0 64s

Zdtp-worker-hfblk 0/1 ContainerCannotRun 0 60s

Zdtp-worker-t9f6v 0/1 ContainerCreating 0 11s

Zdtp-worker-v2g7s 0/1 ContainerCannotRun 0 31s

2) Pod restart policy RestartPolicy=onFailure

When RestartPolicy=onFailure,Pod exits abnormally, Pod attempts to restart until the normal execution of the Pod is completed, and Job will not restart a Pod execution, as shown below:

$kubectl get pod-namespace zdtp-namespace

NAME READY STATUS RESTARTS AGE

Zdtp-worker-5tbxw 0/1 CrashLoopBackOff 5 67s

How to choose K8s scheduling strategy?

After studying the scheduling scheme of K8s and the restart strategy of Pod, we need to choose the appropriate scheduling mode according to the scheduling needs of the project.

1. Incremental synchronization Worker

Incremental synchronization Worker will continue to synchronize without stopping, which means that the restart policy of Pod must be RestartPolicy=Always, so you can only choose Deployment scheduling or directly create Pod deployment. But it is recommended to use Deployment. It has been officially stated that even if the copy of Pod is 1, Deployment is still recommended for deployment.

2. Full synchronous Worker

Full synchronization Worker exits after data synchronization. It seems that Job scheduling or directly creating Pod deployment can be satisfied, but at this stage, since full synchronization does not record synchronization progress for the time being, it is required that any error container cannot restart automatically after exiting. The current practice is that when an abnormal exit occurs during the execution of Worker, users are required to delete synchronized resources by themselves. Then manually start Worker for full synchronization again.

Therefore, Job is not suitable for scheduling Worker Pod, and full synchronous Worker is only suitable for deployment using Pod directly at this stage, and you need to set Pod restart policy RestartPolicy=Never.

After reading the above, do you have any further understanding of how to use K8s for job scheduling? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.