Shulou(Shulou.com)06/03 Report--

Background ConnectionLoss link loss SessionExpired session expiration bypass zookeeper broker status notification leader election and zkNode disconnect do idempotent static expansion, dynamic expansion background

Distributed locks are now used more and more and are usually used to coordinate multiple concurrent tasks. In general application scenarios, there are certain unsafe uses, which may cause multiple master to execute in parallel, and the business or data may have side effects caused by repeated computing, such as playing master without getting the lock.

In order to accurately get the distributed lock and accurately capture the dynamic transfer state of the lock in the distributed case, it is necessary to deal with the chain reaction caused by network changes. For example, the common session expire and connectionLoss, how to ensure that we get the lock accurately when setting the lock status.

When designing tasks, we need to have a stop point strategy, which is a mechanism to deliver execution power after sensing the loss of lock. But whether the problem needs to be dealt with so seriously depends on the business scenario, for example, there is no need to double-count if the downstream tasks are idempotent. But in some cases there is a need for serious and precise control.

ConnectionLoss link lost

Let's start with the first scenario, the connectionLoss event, which indicates that the submitted commit may be executed successfully or failed. Success means that the tcp is disconnected when it is successfully executed in the zookeeper broker, resulting in the failure to get the returned status. Failure means that the link is broken without being submitted to zookeper broker at all.

So we need to do connectionLoss event handling when we get lock. Let's take a look at an example.

Protected void runForMaster () {logger.info ("master:run for master."); AsyncCallback.StringCallback createCallback = (rc, path, ctx, name)-> {switch (KeeperException.Code.get (rc)) {case CONNECTIONLOSS: checkMaster () / / Link failure check whether the znode setting is successful return; case OK: isLeader = true; logger.info ("master:I'm the leader serverId:" + serverId); addMasterWatcher () / / Monitor master znode this.takeLeadership (); / / execute leader rights break; case NODEEXISTS: isLeader = false; String serverId = this.getMasterServerId (); this.takeBackup (serverId) Break;}}; zk.create (rootPath + "/ master", serverId.getBytes (), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL, createCallback, null) / / create master node} / * check master Loop check * / private void checkMaster () {AsyncCallback.DataCallback masterCheckCallback = (rc, path, ctx, data [], stat)-> {switch (KeeperException.Code.get (rc)) {case CONNECTIONLOSS: checkMaster () Return; case NONODE: runForMaster (); return; default: {String serverId = this.getMasterServerId (); isLeader = serverId.equals (this.serverId) If (BooleanUtils.isNotTrue (isLeader)) {this.takeBackup (serverId);} else {this.takeLeadership ();}} return }; zk.getData (masterZnode, false, masterCheckCallback, null);}

The master here means that it has the right to execute, and only if you successfully get the role of master can you fulfill the right of master.

Once the runForMaster method finds a connectionLoss, it initiates the checkMaster to check, and the checkMaster method also does the connectinLoss check until the clear state is obtained. At this point, it is possible that another node has acquired the master role, so the current node is ready to backup and wait for the opportunity.

We need to capture all the state changes in zookeeper, to know when master expires and to be ready to apply, and to release master rights when the session expires when we are master.

/ * Monitoring master znode for master/slave switching * / private void addMasterWatcher () {AsyncCallback.StatCallback addMasterWatcher = (rc, path, ctx, stat)-> {switch (KeeperException.Code.get (rc)) {case CONNECTIONLOSS: addMasterWatcher (); break Case OK: if (stat = = null) {runForMaster (); / / master no longer exists} else {logger.info ("master:watcher master znode ok.");} break Case NONODE: logger.info ("master:master znode delete."); runForMaster (); break;}}; zk.exists (masterZnode, MasterExistsWatcher, addMasterWatcher, null);}

The state monitoring is carried out through the zookeeper watcher mechanism to maintain the linkage with the state changes of the network and zookeeper.

SessionExpired session expires

Let's look at the second question. The first question is how to ensure that the status can be accurately obtained when obtaining the lock, where the status refers to the master role or the backup role.

When we successfully establish a link with zookeeper broker, successfully obtain the master role and are fulfilling our master obligations, zookeeper suddenly notifies session of expiration, and the SessionExpired event indicates that zookeeper will delete all temporary znode created by the current session, which means that the master znode will be created by other sessions.

At this point, we need to hand over our master rights, that is, we must let go of the task at hand, and this stop state must be able to reflect the overall situation. The most common problem at this time is that we are no longer master but are still secretly executing master rights. Through dashboard, we will see a very strange problem that the server that is not master is still executing.

Case SESSIONEXPIRED: / / execute stop point notification this.stopPoint (); break

So here we need to have a stop point strategy, similar to jvm's safe point, to respond to a global stop at any time when designing a task.

Bypass zookeeper broker for status notification

Another common use is to bypass zookeeper for status notification.

We all know that zookeeper cluster consists of multiple instances, each of which is located in different parts of the country or even around the world. There is a great difference in synchronization delay between leader and these nodes. Zookeeper uses a quorum of two-phase submission to complete a commit.

For example, if there are seven instances to form a set of zookeeper cluster, when a client is written to the commit, only more than half of the writes in the cluster need to be completed, even if the commit submission is successful. However, cleint executes the next task immediately after getting the successful response, which may be to read all the state data under a znode, which may not be read at this time.

If it is a distributed lock, it is likely that the transfer of the lock in the zk cluster cannot be maintained with the client cluster. Therefore, as long as it is based on zookeeper for cluster scheduling, the original zookeeper should be used for status notification, and zookeeper cannot be bypassed to schedule rows.

Leader election is disconnected from zkNode

Zookeeper leader is the serializer for all state changes. Add, update, and delete all need to be handled by leader, and then propagated to all follower and observer nodes.

All session is saved in leader, all watcher is saved in zookeper node linked to client, and both scenarios result in untimely notification of state transition.

If zookeeper is a cluster composed of multiple data centers and there is a problem of remote synchronization delay, leader will definitely be placed in the written data center, and zid should be the largest, even a group of high-zid machines are written in the data center, so as to ensure that lead outages will not easily cause leader to be elected to other data centers.

But both follower and observer will have client in use, and there will also be distributed clusters that coordinate on these nodes.

Let's start with the delay perception problem caused by the leader election. For example, the current zookeeper cluster is composed of seven machines:

DataCenter shanghai:zid=100 、 zid=80 、 zid=50dataCenter beijing: zid=10 、 zid=20dataCenter shenzhen:zid=30 、 zid=40

Due to the leader election in the network problem cluster, zid=100 is temporarily separated from the cluster, and zid=80 becomes leader. The old and new log issues are not considered here, and zid is preferred for the election.

Because all the session in the cluster is stored in the original zid=100 machine, the new leader does not have any session information, so all session will be lost.

The holding time of the session depends on the sessinoTimeout time we set. Client propagates the heartbeat to the linked zkNode through ping, which may be the node of any character, and then the zkNode keeps the session through the heartbeat with the zkleaderNode, and the zkNode also maintains the session timeout through ping.

At this point, when client is relinked to zkNode, it will be told sessionExpired. SessionExpired is notified by zkNode, and when a session is lost or expired, client is told by zkNode that the session expires when it tries to link to zkNode.

If client only captures sessionExpired, there will obviously be multiple master runs, because when you are disconnected from zkNode, before you receive the sessionExpired event, another client has successfully created the master to get the rights.

This situation also occurs when the zkNode is detached from the cluster, and the problem of sessionExpired delay notification also occurs after the zkNode is disconnected. All watcher needs to be created on the new zkNode to receive new events.

Static expansion, dynamic expansion

In extreme cases, static expansion may lead to serious data inconsistencies in zookeeper clusters. For example, existing clusters: a, B, C, now need static expansion to stop ABC instances and pull in DE instances. If C instance is the most lagging instance in ABC, if the AB startup speed is not as fast as C, it will cause CDE to form a new cluster, and the new era number will overwrite the original AB log. Of course, static expansion is basically not accepted now, but basically dynamic expansion.

Dynamic capacity expansion can also cause similar problems in extreme cases. For example, there are now three data centers: leader zid=200 in server room 1, 2 and 3, zid=80 in server room 1, zid=80 in server room 100, and zid=40 in server room 50. Assuming that the last commit was submitted between zid=200, 100,50, and 50, the network of server room 1 was cut off, and the zid=80 of server room 2, and the zid=40 of server room 50 and 3 began to form a new cluster, and a new era was generated on zid=50.

Be idempotent

There will be a lot of distributed problems when using zookeeper to implement distributed locking or cluster scheduling. In order to ensure that these problems will not lead to business system or business data inconsistency, we still do a good job of idempotent consideration on these tasks.

For example, do data calculation, do a time check, version check and so on. If it is based on the zookeeper implementation of a separate distributed system will require more work.

Author: Wang Qingpei (Senior architect of Hujiang Group)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.