Shulou(Shulou.com)06/03 Report--

This article introduces the knowledge of "what is the principle of ZooKeeper conversation". In the operation of practical cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Secrets of client session

Conversation, or session, the word or concept is useful in many places. In ZK, conversation means that after two different machines have established a network connection, it can be said that a session has been created between them. The session of ZK has the concept of timeout. When the session times out, the server will actively close it. Of course, the client can also actively request the server to close the session. You may ask, why do you want to make this trouble? why don't you just connect both sides and use it all the time? The concept of session is to prevent some clients from using it infrequently after establishing a connection, and closing the connection early can save resources.

1.1 A day when the chicken is too beautiful

I find that I haven't had a cue chicken for a long time. Let him make his debut in the central position this time.

Our chicken is too beautiful to get up every day, send Weibo, live broadcast, dance, play basketball, a lot of things need to go to the office to handle.

So the first thing to do is to go to the office to find Ma Guoguo (now suppose Ma Guoguo has an office) to apply for the use of the office (establish a connection, create a session)

Ma Guoguo will create an ID for Chicken Taimei, that is, session ID. This ID (I assume 19980802 here) will be bound to Ji Taimei. When Ji Taimei applies, it also needs to tell Ma Guoguo how long the longest timeout is. I assume it is 6000 milliseconds.

And Ma Guoguo will record it here:

At the opening of Ma Guoguo, there is also a session check interval, that is, the tickTime option configured in zoo.cfg, which I assume here is 3000 milliseconds. Ma Guoguo will calculate a timeline when it opens, and the interval of this timeline is fixed and will not change.

Then Ma Guoguo will calculate a chicken Taimei session timeout point by combining Chicken Taimei 6000 and the current timestamp with the timeline.

Then it will be recorded:

After recording, even if Chicken Taimei session is created successfully.

On the other hand, the nodes on Ma Guoguo's side will follow this timeline to check the conversation regularly, assuming that the current time is too beautiful.

Ma Guoguo will take out all the conversations at this point in time (remember what we said above, can it be multiple? )

Then we will find the corresponding villagers according to the ID information and inform them one by one that the session is closed.

You may ask that now because the chicken is too beautiful, the timeout time is 6000, while the horse fruit timeout check is 3000, which is exactly an integer multiple. What if the timeout time is not an integer multiple? Not to say that our Comrade Ma Guoguo is eager to learn and make progress, he has long thought of it, so he has designed an algorithm that will round down to find the checkpoint set by Ma Guoguo, no matter what the villagers' overtime time is.

Suppose the timeout of the chicken is 5900.

For example, the overtime time of chicken beauty is 6500.

So you see, take 3000 of Ma Guoguo as an example, as long as those less than 3000 are calculated according to 0, those less than 6000 are calculated according to 3000, those less than 9000 are calculated according to 6000, and so on, so as long as Ma Guoguo's own inspection interval is determined, no matter what kind of timeout is set by any villager, it can be rounded down to the nearest unified checkpoint. In this way, Ma Guoguo will not have too much burden during the inspection, and the overtime time of the villagers can be checked uniformly.

However, this will definitely result in an error in the timeout of the client (usually a little shorter than set). The way to reduce this error is to reduce the check interval of Ma Guo fruit, that is, the tickTime parameter (the default is 2000, which is enough for me).

And Ma Guoguo's session management will not be the only one who is too beautiful. Let's take a look at what the session management page of multiple villagers looks like.

You can see that three hash tables are used to record these mappings, as shown in the timeline

So when the time reaches 25317000, the corresponding three villagers will time out, and at 25320000, the other two villagers will time out.

I also have to say here that after the session ID opens in Maguoguo, it will initialize a cardinality based on the current time stamp and myid. For example, it may be a number like 987434245. After that, when each villager comes to assign a session ID, he will only add one to this number, so there will be no messy numbers. The figure in the figure is just my personal hobby. It is not in accordance with the actual situation.

But in that case, isn't the chicken too beautiful to time out 6000 milliseconds each time? Of course, this is impossible, because every arbitrary operation of the villagers (add, delete, modify and check) will refresh the timeout timestamp, how exactly do it? Let's take a look at the red arrow. Suppose the red arrow is the timeout calculated by Ma Guoguo for Chicken Tai Mei when the session was first created. Suppose Chicken Tai Mei performs any operation at the place of the green arrow timestamp.

Ma Guoguo will recalculate the new timeout based on the current timestamp (at the green arrow) and the timeout set before the chicken is too beautiful (6000):

Then modify the data on the session management page, and I will still explain it with the examples of several villagers.

Before updating:

After update:

This update process can be called session activation.

1.2 heartbeat detection

For a moment, in addition to refreshing the timeout for each normal operation of the client, the client still needs a mechanism to maintain the session, which is the heartbeat detection we have heard of. the principle is that every time the client starts, it also sets an interval for heartbeat detection. In the backend, we will always judge whether the last timestamp and the current time have exceeded the interval between the heartbeat detection. If so, a request named PING will be sent. As we just said, any operation of the client will refresh the timeout, and PING is no exception. With this heartbeat mechanism, the client can maintain the session state with the server. When the server receives the PING, in addition to the refresh timeout, it simply replies a PING to the client, while the client receives the PING from the server and discards it without any other operation.

Let's take the Java client as an example

ZooKeeper client = new ZooKeeper ("127.0.0.1 purl 2181", 12000, null)

Assuming that the timeout is set to 12000 milliseconds, the heartbeat interval of the client is 4000 milliseconds. The calculation process is as follows

12000 * 2 / 3 / 2 = 4000 / / this formula is the write-dead logic in the code, which is actually / 3

Therefore, as long as the idle time of the client exceeds 4000 milliseconds, a PING will be sent to the server. If the client sets a very large timeout, such as half an hour, it will be forced to send a PING every 10 seconds (this 10 seconds is the logic written by the Java client).

This is the end of the conversation between the client and the server, and then let's talk about the conversation between the server.

2. Secrets of server-side conversations

If the village had multiple offices at the same time (I'll assume two here), the situation would be different.

Suppose Ji Taimei found Ma Xiaoyun as Follower when she first connected:

Follower cannot handle non-read requests alone, so after assigning ID to Ji Taimei, Ma Xiaoyun will forward the session creation operation to Ma Guoguo, just as Chicken Taimei finds Ma Guoguo. The process is the same as above and recorded in the session management page.

Ma Xiaoyun himself will simply maintain a mapping relationship between session ID and timeout, taking multiple villagers as an example, recording them every time they receive a request.

Now Chicken Taimei is connected to Ma Xiaoyun's office (including each heartbeat), but the global session management data is here in Ma Guoguo, so how can the session state be maintained?

Here we have to talk about how the heartbeat occurs between servers.

The server has an important configuration tickTime (default is 2000) and another important configuration syncLimit (default is 5). Let me take these two default values as examples:

First of all, Leader will initiate PING requests to each Follower at a frequency of 1000 (tickTime / 2) milliseconds.

Check whether the timeout period of the PING returned by Follower exceeds 10000 (tickTime * syncLimit) each time, and close the socket connection with the Follower without receiving an ACK response from the Follower.

After receiving the message from PING, Follower will reply a PING to Leader and send the recorded session mapping relationship together.

Will also immediately clear their own local mapping relationship!

Then, after Leader receives the PING response from Follower, because all the session management data of the client are actually here in Leader, Leader can activate the session ID and timeout sent by Leader. The specific method is the same as in the previous example. Through the PING between servers, you can not only detect the heartbeat between servers, but also activate the client session, and eat two fish at a time.

A brief summary:

Session is an important concept in ZK. The state of the session will affect how the server handles the client request.

Each operation of the client extends the timeout of the session, and the client initiates a PING request to maintain the session, so as to avoid being closed by the server when the session timeout is idle.

The session data of the client is saved on the Leader side, and Follower simply records the mapping relationship between the session ID and the timeout time during each operation.

The heartbeat PING between servers is initiated by Leader to Follower actively.

After receiving PING, Follower will send its saved session mapping data to Leader.

After receiving the PING response from Follower, Leader will activate the session data sent.

Now that we know the concept of session, we can talk about temporary nodes.

III. Temporary nodes

Let's first take a look at the code for creating temporary nodes.

Client.create ("/ HelloZooKeeper/niubi", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL)

This creation operation is no different from other persistent node creation. You need to write down a record in a small red book, and one field in this record is ephemeralOwner. When the node is a persistent node, the value of this field is 0, but when the node is a temporary node, this field records the session ID that holds the node.

In addition to creating a record on the little red book, because it is a temporary node, you also need to record it in a special place. Suppose Ji Taimei has created 3 temporary nodes:

19980802 = > [/ the chicken is too beautiful / I am so beautiful "," / the chicken is too beautiful / I am so handsome "," / the chicken is too beautiful / I really show "]

When the chicken Taimei session times out, it may be that the session really timed out (it is not possible because of the heartbeat mechanism), or it may be that the chicken Taimei actively closed the session.

Ma Guoguo will take out the path of the corresponding temporary node from the place where the temporary node is recorded according to the session ID of Ji Tai Mei, and then delete it according to the path. The effect is the same as that of Ji Tai Mei to delete the temporary node actively, so that the corresponding temporary node will be automatically removed when the client is closed. The feature of this temporary node will be used when ZK implements distributed locks, preventing the client from being unable to perform the logic of releasing the lock due to unexpected exit!

IV. Agreement

There is another thing I have never mentioned, which is the agreement of ZK.

As we all know, ZK is an application of CS architecture, which can be divided into client and server, so it is inevitable to carry out network communication, and not only between client and server, but also between server and server. With network communication, protocol is inseparable, but protocol is not only the most important thing, but also the least important thing.

The most important reason is that ZK itself communicates based on this protocol, no matter between the client and the server, the various codes I mentioned earlier, such as REQUEST, ACK, COMMIT, PING and so on. All belong to a field in the protocol that distinguishes different messages. The protocol constitutes the basis of the whole ZK communication, and the function of the whole component can be completed only after being able to communicate.

The least important is because, unless you want to develop a ZK client and take the initiative to request the ZK server, even if you do not know the specific format of the agreement, it will not affect your understanding of the whole ZK principle, and the introduction of the protocol is very boring and useless, easy to dissuade.

So I saved this concept for last, and I'm not going to explain what the different requested protocols in ZK look like. This time I will briefly introduce the agreement from a different point of view.

First of all, the ZK I introduced are Java programs, both client and server, so the essence of the protocol is to specify how to convert Java objects into byte streams to facilitate transmission in the network, and how to convert this byte stream back to Java objects by the party who gets the byte stream, which is actually the process of serialization and deserialization. In order to facilitate serialization, the various objects defined in ZK, such as XxxRequest, XxxResponse, XxxPacket, and so on, usually have only several field types: int, long, String, byte [], List, boolean, and other nested types.

4.1 int 、 long 、 boolean

For these three types, it is the easiest to write directly with the output stream. The difference is that one is 4 bytes, one is 8 bytes, and the other is 1 bytes.

4.2 String 、 byte []

The two are similar. If the field is empty, write a-1. If it is not empty, write an int to represent the length, followed by byte [] to indicate the specific data.

4.3 nested types, List

Encounter List and 4.2 is the same, if empty to write-1, not empty to write List length first, and then traverse List according to generics (which can only be the above) to decide how to continue to write, nested objects, then the write operation will be delegated to it, because its fields can only be the above.

4.4 Summary

The serialization protocol of ZK uses a compact writing method, which can be written into the final byte stream according to different field types.

This is the end of "what is the principle of ZooKeeper conversation"? thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.