Shulou(Shulou.com)06/01 Report--

This article shows you how to implement kubernetes scheduler backend scheduling, concise and easy to understand, absolutely can make your eyes shine, through the detailed introduction of this article I hope you can gain something.

Background As k8s is getting hotter and hotter, as well as automatic deployment, automatic scaling and other advantages, today we will discuss how to implement component diagrams based on k8s backend scheduling.

component description

The whole data flow is the consumer-producer model.

KubernetesClient interacts with k8s, e.g.: task submission, kill task podsPollingSnapshotSource pulls pod task status from k8s, stores it in podSnapshotStorepodsWatchSnapshotSource monitors task watcher to get task status, stores it in podSnapshotStorepodSnapshotStorepod internal status transitionpodsSnapshotpod status mirrortaskPodsLifecycleManager consumes pod status from podSnapshotStore, so as to perform subsequent operations according to task status

specifically stated

For the podsWatchSnapshotSource implementation, we are based on the k8s watch mechanism, but there is a problem:

If at some point, podsWatchSnapshotSource fails and causes the component to restart, then the problem comes, and the event will be lost during the restart.

Here we use the resourceVersion mechanism of k8s. If we store the resourceVersion regularly and read it at the time of restart, we can achieve the effect of breakpoint continuation.

One thing to note is that the resourceVersion is limited in its retention on Kubernetes servers. Old clusters using etcd2 can retain up to 1000 changes.

By default, newer clusters using etcd3 retain changes for the last 5 minutes if the resourceVersion exceeds the server's resourceVersion value

You'll get an error.

data flow diagram

process description

backend Retrieves backend resources that can be retrieved by calling reviveOffer.

After obtaining the resource, submit the task to k8s through kubernetesClient.

Reduce the amount of resources required to submit tasks to k8s

Update the corresponding job status inside backend to Running status. If the existing job status is Runnnig status, update the corresponding job status to updated status.

podsWatchSnapshotSource monitors the task submitted just now, obtains the status of the task update, and stores it in podSnapshotStore for subsequent task processing

podsPollingSnapshotSource regularly pulls all tasks submitted by the app and stores them in podSnapshotStore for final task cleanup

PodSnapshotStore updates task status to internal status and makes function callbacks for snapshots subscribed to this podSnapshotStore

taskPodsLifecycle Manager subscribes to the snapshot above and processes it:

1. If the task status is podFailed or PodSucceeded, update the internal pig status of the backend job; if there is a corresponding Running job, call k8s api to delete the pod and delete the resources (cpus, mem, etc.) occupied by the pod; if there is a corresponding updated job status, update the updated status to Running status, so as to prevent the update of external tasks from causing inconsistent update of task resources.

2. Call the statusUpdate method of kubernetesTaskScheduler Backend to update the task for processing

UML Class Inheritance Diagram

Difference between Spark on K8S

Because the company has its own scheduling platform, it is mainly compared from the granularity of scheduling:

Spark on k8s scheduling is executor-level, coarse-grained scheduling

k8s backend scheduling is job level, each job has a pod container, belonging to fine-grained precise scheduling

The above is how to implement kubernetes scheduler backend scheduling. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserves, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.