Shulou(Shulou.com)06/02 Report--

In large-scale distributed systems, there are bound to be a large number of concurrent writes. In this scenario, how to carry out better concurrency control, that is, to ensure data consistency when multiple tasks access data at the same time, has become a problem that must be solved in distributed systems.

Pessimistic concurrency control and optimistic concurrency control are the main technical means used in concurrency control, and different control methods should be chosen for different business scenarios.

Pessimistic lock

Pessimistic concurrency control (also known as pessimistic lock, Pessimistic Concurrency Control, abbreviation "PCC") is a method of concurrency control. It prevents one transaction from modifying data in a way that affects other users. If the operation performed by a transaction reads that a row of data has a lock applied, other transactions can perform operations that conflict with the lock only when the transaction releases the lock.

In the pessimistic lock scenario, suppose that users An and B want to modify the same file. In the process of locking and modifying the file, B cannot modify the file. B can not acquire the lock until A completes the modification and releases the lock. Then modify the file. It can be seen that pessimistic lock is pessimistic about the control of concurrency. Before making any modification, pessimistic lock will first lock it to ensure that there will be no conflict in the whole modification process, so as to effectively ensure data consistency. However, this mechanism also reduces the concurrency of the system, especially when there is no conflict between the two objects modified at the same time. At the same time, deadlocks may occur during competitive locks, so many systems such as Kubernetes adopt optimistic concurrency control methods.

Optimistic lock

Optimistic concurrency control (also known as "optimistic lock", Optimistic Concurrency Control, abbreviation "OCC") is a method of concurrency control. It assumes that multi-user concurrent transactions will not affect each other and that each transaction can process its own data without requesting a lock. Before committing the data update, each transaction checks to see if any other transaction has modified the data after the transaction has read the data. If other transactions are updated, the transaction being committed will be rolled back.

Compared with the advance control of pessimistic locks, optimistic locks believe that the probability of conflicts between requests is relatively small, is unlocked in the process of reading and changing, and conflicts are detected only when updates are finally submitted, so they have an absolute advantage in systems with high concurrency. Also suppose that users An and B want to modify the same file, An and B will first get the file locally, and then modify it. If A has been modified and the data is submitted, and B submits again at this time, the server will inform B that the file has been modified and return a conflict error. At this point, the conflict must be resolved by B. you can retrieve the file, modify it again and submit it.

Optimistic locks usually determine whether the request conflicts by adding a resource version field. Specify a version value when initializing, read the version number together each time you read the data, update the data each time, and update the version number at the same time. When the server receives the data, compare the version number in the data with that on the server. If it is inconsistent, the data has been modified and a conflict error is returned.

Concurrency Control in Kubernetes

In the Kubernetes cluster, the frequent data update operations of external users and internal components lead to a large amount of concurrent data reading and writing in the system. It is assumed that the pessimistic parallel control method will seriously damage the cluster performance, so Kubernetes adopts the optimistic parallel control method. Kubernetes implements optimistic concurrency control by defining the resource version field, which is contained in the metadata (Metadata) of the Kubernetes object. The resource version (ResourceVersion) field is contained in the Kubernetes object. This string-formatted field identifies the build number of the object, its value comes from the modifiedindex of etcd, and when the object is modified, the field is modified accordingly. It is worth noting that this field is maintained by the server and is not recommended to be modified on the client side.

Type ObjectMeta struct {

.

/ / An opaque value that represents the internal version of this object that can

/ / be used by clients to determine when objects have changed. May be used for optimistic

/ / concurrency, change detection, and the watch operation on a resource or set of resources.

/ / Clients must treat these values as opaque and passed unmodified back to the server.

/ / They may only be valid for a particular resource or set of resources.

/ /

/ / Populated by the system.

/ / Read-only.

/ / Value must be treated as opaque by clients and.

/ / More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#concurrency-control-and-consistency

/ / + optional

ResourceVersion string

.

}

Kube-Apiserver can use this field to determine whether the object has been modified. When the update request containing ResourceVersion arrives at Apiserver, the server will compare the request data with the resource version number of the data in the server. If it is inconsistent, it indicates that the server object has been modified when the update is submitted, and Apiserver will return a conflict error. The client needs to obtain the server data again, modify it and submit it to the server again. The above parallel control method can prevent the following data race:

Client # 1: GET Foo

Client # 2: GET Foo

Client # 1: Set Foo.Bar = "one"

Client # 1: PUT Foo

Client # 2: Set Foo.Baz = "two"

Client # 2: PUT Foo

When concurrency control is not used, assuming that the above request sequence occurs, two clients obtain the same object Foo (including Bar and Baz fields) from the server at the same time. Client#1 first sets the Bar field to one, and then when Client#2 requests to update the Baz field assignment to the server, the modification of Bar by Client#1 will be overridden. Instead, add the resource version field to the object, and the same request sequence will be as follows:

Client # 1: GET Foo / / initial Foo.ResourceVersion=1

Client # 2: GET Foo / / initial Foo.ResourceVersion=1

Client # 1: Set Foo.Bar = "one"

Client # 1: PUT Foo / / Update Foo.ResourceVersion=2

Client # 2: Set Foo.Baz = "two"

Client # 2: PUT Foo / / returns 409 conflicts

After Client#1 updates the object, the resource version number will change, and Client#2 will return a conflict error (409) when the update is submitted. In this case, Client#2 must reacquire the data locally, update it and then submit it to the server.

Assuming that the Resource version value is not set in the object of the update request, Kubernetes will decide whether to make a hard update based on the hard rewrite policy (configurable). If configured to be hard rewritable, the data will be updated directly and saved in Etcd, otherwise an error will be returned, prompting the user to specify ResourceVersion.

Update and Patch in Kubernetes

Kubernetes implements two object update methods, Update and Patch, which provide different update operations, but the conflict judgment mechanism is the same.

Update

For Update, the client update request contains the entire obj object, and the server compares the Resource version value of the obj object in the request with the latest server-side obj object. If equal, there is no conflict, and the entire object will be updated successfully. On the contrary, if it is not equal, a 409 conflict error is returned. The code snippet for conflict judgment in Kube-Apiserver is as follows.

E.Storage.GuaranteedUpdate (ctx, key...) (runtime.Object, * uint64, error) {

/ / If AllowUnconditionalUpdate () is true and the object specified by

/ / the user does not have a resource version, then we populate it with

/ / the latest version. Else, we check that the version specified by

/ / the user matches the version of latest storage object.

ResourceVersion, err: = e.Storage.Versioner () .ObjectResourceVersion (obj)

If err! = nil {

Return nil, nil, err

}

Version, err: = e.Storage.Versioner () .ObjectResourceVersion (existing)

DoUnconditionalUpdate: = resourceVersion = = 0 & & e.UpdateStrategy.AllowUnconditionalUpdate ()

If doUnconditionalUpdate {

/ / Update the object's resource version to match the latest

/ / storage object's resource version.

Err = e.Storage.Versioner () .UpdateObject (obj, res.ResourceVersion)

If err! = nil {

Return nil, nil, err

}

} else {

/ / Check if the object's resource version matches the latest

/ / resource version.

.

If resourceVersion! = version {

Return nil, nil, kubeerr.NewConflict (qualifiedResource, name, fmt.Errorf (OptimisticLockErrorMsg))

}

}

.

Return out, creating, nil

}

The basic process is:

Get the ResourceVersion value of the obj object in the current update request and the ResourceVersion value of the latest obj object (existing) on the server. If the ResourceVersion value of the obj object in the current update request is equal to 0, that is, the client has not set this value, it is determined whether to hard overwrite (AllowUnconditionalUpdate). If configured as a hard rewriting policy, the obj object will be updated directly. If the ResourceVersion value of the obj object in the current update request is not equal to 0, then determine whether the two ResourceVersion values are consistent. Inconsistency returns conflict error (OptimisticLockErrorMsg)

Patch

Compared to the Update request, which contains the entire obj object, the Patch request implements a finer-grained object update operation, which contains only the fields that need to be updated. For example, to update the image of container in pod, use the following command:

Kubectl patch pod my-pod-p'{"spec": {"containers": [{"name": "my-container", "image": "new-image"}]}'

The server only receives the above patch information, and then updates the patch to the Etcd with the following code.

Func (p * patcher) patchResource (ctx context.Context) (runtime.Object, error) {

P.namespace = request.NamespaceValue (ctx)

Switch p.patchType {

Case types.JSONPatchType, types.MergePatchType:

P.mechanism = & jsonPatcher {patcher: P}

Case types.StrategicMergePatchType:

SchemaReferenceObj, err: = p.unsafeConvertor.ConvertToVersion (p.restPatcher.New (), p.kind.GroupVersion ())

If err! = nil {

Return nil, err

}

P.mechanism = & smpPatcher {patcher: P, schemaReferenceObj: schemaReferenceObj}

Default:

Return nil, fmt.Errorf ("% v: unimplemented patch type", p.patchType)

}

P.updatedObjectInfo = rest.DefaultUpdatedObjectInfo (nil, p.applyPatch, p.applyAdmission)

Return finishRequest (p.timeout, func () (runtime.Object, error) {

UpdateObject, _, updateErr: = p.restPatcher.Update (ctx, p.name, p.updatedObjectInfo, p.createValidation, p.updateValidation, false, p.options)

Return updateObject, updateErr

})

}

The basic process is:

1. First, judge the type of patch and select the corresponding mechanism according to the type.

two。 Use the DefaultUpdatedObjectInfo method to add applyPatch (the method of applying Patch) to the header of admission chain

3. Finally, the above Update method is called to perform the update operation.

In step 2, hang the applyPatch method to the head of the admission chain, similar to the admission behavior, the applyPatch method applies the patch to the newly acquired server-side obj, generates an updated obj, and then continues to execute the Admit and Validate in the admission chain on the obj. In the end, the update method is called, so the mechanism of conflict detection is exactly the same as the Update method above.

The main advantage over Update,Patch is that the client does not have to provide full amount of obj object information. The client only needs to submit the field information to be modified in the way of patch, and the server will apply the patch data to the newly acquired obj. The steps of obtaining, modifying and submitting the full obj on the client side are omitted, which reduces the risk of data modification and greatly reduces the probability of conflict. Because Patch method has absolute advantages in transmission efficiency and collision probability, Patch method is used in almost all update operations in Kubernetes. We should also pay attention to using Patch method when writing code.

Attached:

The ResourceVersion field is used not only in the concurrency control mechanism mentioned above in Kubernetes, but also in the list-watch mechanism of Kubernetes. The list-watch on the Client side is divided into two steps: first, list retrieves all objects, and then incrementally watch subsequent objects. After retrieving all objects from list, Client will use the ResourceVersion of the latest object as the starting parameter for the next watch operation, that is, Kube-Apiserver returns subsequent data with the received ResourceVersion as the starting point, ensuring the continuity and integrity of the data in list-watch.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.