Shulou(Shulou.com)05/31 Report--

This article shows you the Botposter.com cluster ETCD2.3.7 upgrade to 3.0 example analysis, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Botposet.com is a marketing automation SAAS product similar to HubSpot, all developed in golang.

Description

In Botposter.com, ETCD is mainly used for the following two responsibilities:

Master election

Cluster information preservation

Early on, ETCD's TTL was used to implement master heartbeat detection, which was removed from Botposter.com 's refactoring last month for performance reasons. This also makes it easier to upgrade because ETCD v3 has a major change to TTL.

Prepare the data.

The main reference for the migration work is as follows:

Https://github.com/coreos/etcd/blob/master/Documentation/op-guide/v2-migration.md

Https://github.com/coreos/etcd/blob/master/Documentation/upgrades/upgrade_3_0.md

Test environment

Before starting the upgrade, you need to build a test environment, the process is very simple, which ETCD does very well, V3 version and V2 version are exactly the same in terms of installation and configuration parameters.

Installation reference link: https://github.com/coreos/etcd/releases

Configuration reference link: https://github.com/coreos/etcd/blob/master/Documentation/op-guide/clustering.md

After the test environment is configured, use etcdctl to test whether ETCD V3 is working properly. It is important to note here that you must not forget to add ETCDCTL_API=3 to the environment variable. Otherwise, when operating on V3, there is no data return or error return regardless of using SET,GET. It is recommended that ETCD V3 can provide an error prompt. I wasted some time here because I took it for granted that using ETCD V3 and ETCDCTL V3 would match by default. Document link for ETCDCTL: https://github.com/coreos/etcd/blob/master/etcdctl/README.md#migrate-options

Note: the first time I use the etcdctl member list command (all commands go wrong, here I use member list as an example), I return the following error code:

Grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 127.0.0.1" 22379: getsockopt: connection refused "; Reconnecting to {" 127.0.0.1 "}

Running the etcdctl command alone returns help for using etcdctl, with one line:

-- endpoints= [127.0.0.1VOR 2379127.0.1RU 22379127.0.1RU 32379] gRPC endpoints

Originally, there are three endpoints for default gRPC, but there are two known ways to solve this problem: one is to add the-- endpoints parameter to the etcdctl command line.

Etcdctl-endpoints=127.0.0.1:2379 member list

The second is to add other ports to the etcd startup parameters

-listen-client-urls http://127.0.0.1:2379,http://127.0.0.1:22379,http://127.0.0.1:32379

Use the second method temporarily in Botposter.com. Because of the limited migration time, I did not continue to see if the default of gRPC, endpoints, could be modified.

The difference between API V2 and V3

Transactions: ETCD V3 provides multi-key conditional transactions (multi-key conditional transactions), and various applications need to use transactions instead of the original Compare-And-Swap operations.

Flat key space (Flat key space): ETCD V3 no longer uses the directory structure and retains only keys. For example, "/ a/b/c/" is a key, not a directory. Prefix query is provided in V3 to obtain all the key values that meet the prefix conditions, which in turn realizes the function of querying all subdirectories and nodes under a directory in V2.

Concise response: successful operations such as DELETE will no longer return the pre-operation values. If you want to get the value before deletion, you can use a transaction to implement an atomic operation to get the key value and then delete it.

Lease: the lease replaces the TTL implementation in V2, where the TTL is bound to a lease and the key is attached to the lease. When the TTL expires, the lease is destroyed and the keys attached to the lease are deleted.

The only changes related to Botposter.com are the flat key space, because the system uses the ETCD directory structure to hold all the information about master,node and task. From the presentation of the official documentation, transactions and leases are worth testing and used to optimize the use of V2.

Client code upgrade flat key space

Modify the parts of the original code that contain the following code to

& client.GetOptions {Recursive: true}

Are modified to

Clientv3.WithPrefix () data type

In the V2 version, the data type of resp.Node.ModifiedIndex is int64 in uint64,V3. Because resp.Node.ModifiedIndex is used as the global sequence identity in Botposter.com, the data type in the original system needs to be changed to int64.

Compare-And-Swap

A typical use of V2 is Compare-And-Swap, which is also used in Botposter.com to implement distributed locks by adding the following operations to SET operations:

& client.SetOptions {PrevExist: "false"})

That is, the write can be successful only if the current key does not exist.

In V3, it is implemented by transactions instead. The specific code is as follows:

Kvc: = clientv3.NewKV (& cli) r, _: = kvc.Txn (context.Background ()). If (clientv3.Compare (clientv3.CreateRevision (key), "=", 0). Then (clientv3.OpPut (key, v)). Commit ()

The specific usage of Txn reference: https://godoc.org/github.com/coreos/etcd/clientv3#example-KV--Txn

Node

In V2, the value returned by the get operation response is saved in response.node.value, and if it is a directory, the returned result set is saved in response.node.nodes. V3 has been greatly modified because there is no longer directory in V3, and all key are flat key, so the return values of all get operations are stored in GetResponse.Kvs (data type is [] * mvccpb.KeyValue). And in V2, keynotfound and other errors are no longer retained in V3. In V3, when the key of the query does not exist, the GetResponse.Count is 0Len (GetResponse.Kvs) and the error returned by the 0memery get operation is nil. So the code in V2 is like

Response.Node.Value

Need to be changed to

GetResponse.Kvs [0] .Value

It is also worth noting that the return values of key and value in V3 are of type [] byte, which reduces a lot of data type conversion operations between string and [] byte.

ETCD upgrade

ETCD upgrade is easy, first follow the installation reference link: https://github.com/coreos/etcd/releases, download and extract the file. Because the Bostposter.com cluster has an automatic recovery mechanism, run scripts on all servers using offline upgrades:

Service etcd stopcp etcd / usr/local/binservice etcd start

All startup parameters of ETCD do not need to be modified, and the upgrade time is less than 1 second. After the ETCD upgrade, the code for upgrading the cluster service only needs to restart more than 2000 processes when upgrading the process container, and the recovery time is about 1 minute.

At this point, the upgrade work has been completed. The system function and cluster are tested, and there are no problems.

The following is about the feeling after upgrading to ETCD V3. Time is limited and accurate testing has not been done. Without data support, it is slightly less rigorous.

First, the V3 server side takes up more memory than V2, at least 50% higher. Especially when the pressure increases, the memory footprint soars quickly, and the memory will be released a few minutes after the pressure decreases.

Second, be sure to Close after using Client, because in V2, sync.pool is used in Botposter.com to save Client. When upgrading to V3, pooled Client takes up a lot of memory when the operation is frequent, because it is not clear how much memory a Client takes up without specific testing. The current solution is that Client is no longer pooled and Close immediately after use.

Third, the API of V3 is more reasonable, and the direct result is that the amount of code is reduced and exception handling becomes simpler.

Fourth, from the overall performance after the upgrade, the performance of V3 is much better than V2.

Overall, I recommend upgrading to ETCD V3 if possible.

The above content is the Botposter.com cluster ETCD2.3.7 upgrade to 3.0 instance analysis, have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.