Shulou(Shulou.com)06/02 Report--

As a distributed system, ES needs the cooperation of multiple nodes to manage the cluster and handle user requests. So there is a natural question, how is the ES cluster formed?

The so-called cluster is that multiple computers work together. Since it is collaborative work, it must be in step and step in order to be liberated. A leader role is needed to coordinate resources, which is named Master in ES. The role of Master is not unique to ES. Almost all distributed systems have this role, such as Zookeeper, Mongo and so on.

The mechanism by which Master is produced is also interesting: elections. Since it is an election, there is bound to be a question: how to elect? This is the so-called "election algorithm". The most famous of the "election algorithms" is the Paxos algorithm, which is also the most difficult to understand. Fortunately, ES does not use such a complex algorithm, ES uses the Bully algorithm. The problem that ES needs to solve is the election of nodes, and the Paxos algorithm not only solves the problem of consistency, but also uses a knife to kill chickens.

Based on Bully algorithm, ES implements ZenDiscovery. By the way, the nodes of ES are found to be handled by a unified module, which is DiscoveryModule. Interested to know the ES source code, can be used as an entry.

The process of ZenDiscovery is quite simple, just two steps:

1. Each node communicates with other nodes, gets the nodeId of other nodes, and chooses the one with the lowest nodeId as its own vote. two。 Each node receives votes from other nodes, and if one node gets enough votes, it accepts the fact that it is a Leader and begins to distribute node status to other nodes in the entire cluster.

The following is to understand this process through specific code. Since you understand the process of building an ES cluster, start with the start of the ES process, taking elasticsearch-2.4.5 as an example.

We know that the startup command for ES is bin/elasticsearch, which calls the main method of org.elasticsearch.bootstrap.Elasticsearch.java.

After starting elasticsearch, you can determine by using the following command:

$jps13201 Elasticsearch$$ cat / proc/13201/cmdline | strings/opt/jdk1.8.0_51/bin/java-Xms256m-Xmx1g-Djava.awt.headless=true-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+UseCMSInitiatingOccupancyOnly-XX:+HeapDumpOnOutOfMemoryError-XX:+DisableExplicitGC-Dfile.encoding=UTF-8-Djna.nosys=true-Des.path.home=/opt/elasticsearch-2.4.5-SNAPSHOT/opt/elasticsearch-2.4.5-SNAPSHOT/lib/elasticsearch-2.4.5-SNAPSHOT.jar:/opt/elasticsearch- 2.4.5-SNAPSHOT/lib/*org.elasticsearch.bootstrap.Elasticsearchstart

Next, a Node object is instantiated to represent the ES node. Then start this node.

Bootstrap.init (args); / / Elasticsearch.java line 45 INSTANCE.start (); / / Bootstrap.java line 288 node.start (); / / Bootstrap.java line 222discoService.joinClusterAndWaitForInitialState (); / / Node.java 286

If you do the following configuration in the es configuration file, you can debug this process.

/ / IP address of the current startup node network.host: 192.168.43.239 / / IP list of the cluster discovery.zen.ping.unicast.hosts: ["192.168.43.239", "192.168.43.239 IP 9800", "192.168.43.239 IP 9900"]

We ignore the jump between codes and go directly to the core business logic code ZenDiscovery.innerJoinCluster (). The specific business logic is as follows:

S1: confirm master ZenDiscovery.findMaster ()

S2: determine whether master is the current node, if so, wait for other nodes to join; otherwise, connect to master, and then initiate a request for status update to master.

There are many other details about the communication between cluster nodes. Let's put aside the knowledge related to ES and return to the operating system level. The communication between two computers depends on the knowledge of computer network. To put it simply, it is the TCP/IP protocol. In terms of the implementation of the Java language, it is Socket programming. Socket programming follows the Cmax S mode, in which one computer acts as a server and listens to a port, while another computer acts as a client to connect to that port. In general, developers do not use native Socket programming, but rather use the Netty framework. The Netty framework encapsulates complicated underlying operations and does a lot of work on performance. Its asynchronous / time-driven characteristics make it the preferred framework for network programming. Based on the Netty framework, ES builds RPC services that are functionally similar to dubbo, which is called Transport. The entry to the code is `TransportModule`. About the details of Transport, you need to write a separate blog description, not the focus of this article, skip. Based on the Transport module, ES builds DiscoveryModule, which is the node discovery of ES. That is, the core point that this article tries to understand. Reference: https://www.jianshu.com/p/9454ac19921dhttps://www.elastic.co/blog/found-leader-election-in-general

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.