Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to achieve distributed coordination Kubernet, Xiaobian thinks it is quite practical, so share it for everyone to make a reference, I hope you can gain something after reading this article.

Processing scheduling Jobs

A single application, previously a single instance running on a single node, consists of a number of scheduled jobs (now also publishing business events) in the database update state. The monomer program is created in java and uses Spring heavily, so the job looks like this:

After Spring, confirm that the doSomethingEveryMinute method mentioned above is executed once a minute. The problem is that if we don't currently host a monolithic program on Kubernetes and run it with multiple instances, the job will be executed once per minute on each instance, not just once per minute. This is a problem if jobs have side effects like sending notification emails or updating databases. So how do we avoid this? Of course, there are many solutions, and the obvious choice is to take advantage of Kubernetes Jobs and let Kubernetes schedule jobs periodically. The problem is that this function is only available in Kubernetes version 1.3 and above, but 1.3 has not yet been released. But even if we were able to use such a feature, it would not be technically feasible. Our jobs are highly coupled to an existing code base and extract each job into its own application, which can be buggy and time-consuming if done in one go. So our initial plan was to extract all scheduled jobs into an application that could only run as an instance in Kubernetes. But because of the nature of existing code, and the high coupling, even that is difficult to implement. So, is there an easy way to keep jobs in a standalone app for now, and gradually replace them as we extract functionality from that app into standalone services? Actually, there is.

Leader elections in Kubernetes

To solve this, we need to do some distributed coordination. For example, when jobs are executed by Spring, if this node is not the "leader node" responsible for running the scheduling jobs, we only need to send back information (and don't run the code with jobs). There are projects that help us deal with things like zookeeper and hazelcast. But setting up and maintaining zookeeper clusters just to decide which nodes perform scheduling jobs is too much work. We need something manageable, what if we could leverage Kubernetes? Kubernetes has processed leader elections under cover (using the RAFT consensus algorithm). It turns out that this functionality has been exposed to end users through the use of the gcr.io/google_containers/leader-elector Docker image. There's already a great blog post describing in great detail how this works, click here: http://blog. kubernetes. io/2016/01/simple-leader-election-with-kubernetes.html. So I won't go into details here, but I'll talk about how we use mirroring to solve our problems.

solve problems

What we did was bring the gcr.io/google_containers/leader-elector container to our pod so that each instance of the singleton app could run an instance of the leader election. This is a classic example of how Kubernetes pods work.

Here is an excerpt from a pod defined in our configuration resource:

We launch leader elections and our mono-app. Note that we take--election=monolith-jobs as the first argument. This means that the leader election knows which group the container belongs to. So the containers assigned to this group will be part of the leader election process, and only one container from this group will be elected leader. The second argument to--http=localhost:4040 is also very important. It opens a web server on port 4040, where we can query the pod name of the current leader and return it in this format:

This is a trick we use to decide whether or not to run our job. All we have to do is check if the name of the upcoming dispatch pod matches the elected leader, and if so, we should go ahead and execute the job, or whatever else we should return. For example:

So let's see how ClusterLeaderService is implemented. First, we have to get the pod name from the app. Kubernetes stores the pod name in/etc/hostname, and Java exposes this/etc/hostname to the HOSTNAME environment variable, which we will refer to in this example. Another method is to use the Downward API to expose pod names to environment variable selection. For example:

From here, we can see that metadata.name (i.e. the name of the pod) is associated with the MY_POD_NAME environment variable. But now let's see what the implementation of ClusterLeaderService looks like:

In this example, we are using JsonPath from the RESTAssured project to query the Voter Web Service and extract the pod name from the response. Then we simply compare the name of the local container to leader, and if they are the same, then we know that this instance is leader! That's it!

conclusion

It turns out that the above work works very well! If the leader node dies, another one will be elected automatically. But this process takes a while, about a minute. So it's a trade-off. If your job does not allow you to miss any job execution, then this option is not appropriate for you. But in our example above, there is a minute in the middle where the job is not executed accurately, which is harmless to us. So I think this approach is pretty simple and valuable when porting an existing application that includes scheduling jobs, because scheduling jobs is always hard to extract for a variety of reasons.

About "how to achieve distributed coordination Kubernet" this article is shared here, I hope the above content can be of some help to everyone, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.