Shulou(Shulou.com)06/02 Report--

This article mainly explains "what is the design and implementation of the lock mechanism in Kubernetes". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the design and implementation of the lock mechanism in Kubernetes"?

The basic part of locking for final state

In distributed systems, there are usually a variety of locks. Let's first take a look at what the mainstream locks have in common and how they are designed.

Locks in distributed systems

There are many ways to implement locks in distributed systems: based on CP model and based on AP model, but these locking mechanisms all have some general design principles, so let's take a look at this part first.

1. Lock voucher

The lock certificate is mainly used to prove who holds the lock, and the implementation varies in different systems, for example, it is a temporary sequential node in zookeeper, but it is composed by uuid+threadID in redission, and LeaderElectionRecord is used in K8s to identify which client added the lock.

two。 Lock timeout

When the leader node holds the lock, the rest of the node needs to try the competitive lock, which is usually maintained by the server in the CP system, that is, if the corresponding node is found to have no heartbeat, the node will be kicked out and callback through the watch mechanism, while in the AP system, the client needs to maintain it, such as the timestamp in redission.

3. Clock

In distributed systems, we usually can not guarantee that the physical clocks of each node are completely consistent, and there is usually a concept of logical clock. In many systems, such as raft and zab, it is actually an incremental global counter, but in redission it is through the physical clock, that is, we need to ensure that everyone's physical clock is synchronized as much as possible, and can not exceed the lock timeout.

Network partitioning problem

Whether it is CP or AP, we usually have to ensure the availability of P-that is, partition in the distributed system, so if the Leader node holding the lock has the case of network partition, we need a protection mechanism, that is, the Leader node needs to exit actively.

In zookeeper, because leader nodes need to maintain heartbeats through session, if the corresponding leader nodes are partitioned, session will not be able to perform heartbeats and will exit, so we need to inform our main process to clean up the exit.

Implementation Mechanism of Resource Lock

In fact, resource locking can be realized by manipulating a resource (sequential consistency), with the help of the idea of locking mentioned above. The first core process is as follows:

Store lock credential information through resource objects

Put the information that identifies the current Leader node into the corresponding credential, and try to compete for locks and attempt to acquire locks.

Lock timeout

The lock timeout mechanism of K8s is more interesting, that is, it does not care about your logical clock, but is based on the local clock, that is, each node will store the time when the change of the leader node is observed, and then detect whether to re-initiate the leader competition according to the local lock timeout.

Core source code analysis

Because of the space, only configMap-based resourceLock is introduced here, and the others are more or less the same.

LeaderElectionRecord

In my understanding, the design of this number structure is the real lock (just like in life we can buy a lock and lock all kinds of doors). The implementation details of the system are masked by this lock, but note that this lock is not a strict distributed mutex.

Data structure

In the implementation of the lock, the data is mainly divided into three categories: identity credentials, timestamps, global counters, and then we guess the corresponding design ideas in turn.

Type LeaderElectionRecord struct {HolderIdentity string `json: "holderIdentity" `LeaseDurationSeconds int `json: "leaseDurationSeconds" `AcquireTime metav1.Time `json: "acquireTime" `RenewTime metav1.Time `json: "renewTime" `LeaderTransitions int `json: "leaderTransitions" `}

Identity credential: HolderIdentity

Identity certificate is mainly used to identify a node information. In some distributed coordination systems, it is usually a mechanism that comes with the system, such as session in zookeeper. In the case of resource locking, it is mainly used to verify whether the current node has acquired the lock in the subsequent process.

Timestamp: LeaseDurationSeconds, AcquireTime, RenewTime

Because of the time synchronization problem mentioned earlier, the time correlation here is mainly used for leader nodes to trigger node changes (the Lease type is also in use), while non-Leader nodes detect whether the leader node is alive according to whether the current record has changed.

LeaderTransitions

Counter is mainly through counting to record the number of leader node switching.

ConfigMapLock

The so-called resource lock is actually through the creation of a ConfigMap instance to save our lock information, and through the maintenance of this instance information to achieve lock competition and release.

1. Create a lock

By taking advantage of the idempotent operation of etcd, it can be guaranteed that only one leader node will successfully create the lock at the same time, and the lock will be submitted through Annotations to submit the LeaderElectionRecord mentioned above.

Func (cml * ConfigMapLock) Create (ler LeaderElectionRecord) error {cml.cm, err = cml.Client.ConfigMaps (cml.ConfigMapMeta.Namespace). Create (& v1.ConfigMap {ObjectMeta: metav1.ObjectMeta {Name: cml.ConfigMapMeta.Name, Namespace: cml.ConfigMapMeta.Namespace, Annotations: map [string] string {LeaderElectionRecordAnnotationKey: string (recordBytes),},} }) return err}

two。 Acquire lock

Func (cml * ConfigMapLock) Get () (* LeaderElectionRecord, [] byte, error) {cml.cm, err = cml.Client.ConfigMaps (cml.ConfigMapMeta.Namespace) .Get (cml.ConfigMapMeta.Name, metav1.GetOptions {}) recordBytes, found: = cml.cm.Annotations [LeaderElectionRecordAnnotationKey] if found {if err: = json.Unmarshal ([] byte (recordBytes), & record) Err! = nil {return nil, nil, err}} return & record, [] byte (recordBytes), nil}

3. Update lock

Func (cml * ConfigMapLock) Update (ler LeaderElectionRecord) error {cml.cm.Annotations [LeaderElectionRecordAnnotationKey] = string (recordBytes) cml.cm, err = cml.Client.ConfigMaps (cml.ConfigMapMeta.Namespace) .Update (cml.cm) return err}

LeaderElector

The core process of LeaderElector is divided into three parts: contention lock, timeout detection and heartbeat maintenance. first, all nodes will compete for resource locks, but in the end, only one node will become a Leader node, and then the core process will be divided into two main processes according to their roles. Let's take a look at its implementation.

1. Core process

If the node does not succeed without acquire, it will keep trying until it is cancelled or the election is successful, while the leader node will execute the callback that becomes the leader node (complements the implementation mechanism of leader-based zookeeper)

Func (le * LeaderElector) Run (ctx context.Context) {defer func () {runtime.HandleCrash () le.config.Callbacks.OnStoppedLeading ()} () if! le.acquire (ctx) {/ / Select Lock return / / ctx signalled done} / / if the lock election is successful, the leader node will perform the remaining process Non-leader nodes continue to try acquire ctx, cancel: = context.WithCancel (ctx) defer cancel () go le.config.Callbacks.OnStartedLeading (ctx) le.renew (ctx)}

two。 Renewal of Lock

If you run for the leader node, you need to renew the lock, that is, by calling the lock update operation mentioned above, periodically update the lock record information, namely LeaderElectionRecord, so as to achieve the goal of renewal.

Func (le * LeaderElector) renew (ctx context.Context) {ctx, cancel: = context.WithCancel (ctx) defer cancel () wait.Until (func () {timeoutCtx, timeoutCancel: = context.WithTimeout (ctx, le.config.RenewDeadline) defer timeoutCancel () err: = wait.PollImmediateUntil (le.config.RetryPeriod, func () (bool, error) {done: = make (chan bool 1) go func () {defer close (done) / / renewal done of lock

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.