Shulou(Shulou.com)06/03 Report--

A distributed service cluster management usually needs a coordination service, which provides service registration, service discovery, configuration management, group service and other functions, while the coordination service itself should be a highly available service cluster. ZooKeeper is a widely used and well-known coordination service. The high availability of coordination service itself needs election algorithm to support. This paper will describe the election principle and take the coordination service NebulaBeacon of distributed service cluster NebulaBootstrap as an example to illustrate the election implementation of coordination service in detail.

Why did    choose NebulaBeacon to illustrate the electoral implementation of the coordination service? On the one hand, I have not read the code of Zookeeper, and on the other hand, the election implementation of NebulaBeacon only has more than 200 lines of code, which is simple and concise and easy to explain. NebulaBootstrap, a distributed service cluster based on high-performance C++ network framework Nebula, is a solution to quickly build high-performance distributed services with C++.

Why should    implement its own coordination service instead of using Zookeeper directly? If you want to build a C++ wheel, the whole cluster is C++ services. Because you choose ZooKeeper, you need to deploy a Java environment, and the configuration is not the same system as other services, so it is not a good choice. Spring Cloud has Eureka,NebulaBootstrap and NebulaBeacon. In the future, NebulaBootstrap will support ZooKeeper. However, there is no timetable, so NebulaBeacon is the first choice.

1. Election algorithm selection

   Paxos algorithm and ZooKeeper ZAB protocol are two well-known election algorithms. ZAB protocol is mainly used to build a highly available distributed data master and standby system, such as ZooKeeper, while Paxos algorithm is used to build a distributed consistent state machine system. There are also many applications that use simple election algorithms designed by themselves, which usually rely on the computer's own factors as election factors, such as IP address, CPU core number, memory size, custom sequence number and so on.

   Paxos specifies four roles (Proposer,Acceptor,Learner, and Client) and two phases (Promise and Accept). The    ZAB service has four states: LOOKING, FOLLOWING, LEADING, and OBSERVING.    NebulaBeacon is a coordination service for highly available distributed systems, and it is more appropriate to use ZAP protocol, but ZAP protocol is still a little complicated. NebulaBeacon's election algorithm implements node-based IP address identification, which is fast and easy to implement.

two。 Election-related data structure

The election-related data structure of    NebulaBeacon is very simple:

Const uint32 SessionOnlineNodes::mc_uiLeader = 0x80000000; / /

< uint32最高位为1表示leaderconst uint32 SessionOnlineNodes::mc_uiAlive = 0x00000007; ///< 最近三次心跳任意一次成功则认为在线std::map m_mapBeacon; ///< Key为节点标识，值为在线心跳及是否为leader标识   如上数据结构m_mapBeacon保存了Beacon集群各Beacon节点信息，以Beacon节点的IP地址标识为key排序，每次遍历均从头开始，满足条件（1&&2 或者 1&&3）则标识为Leader：1. 节点在线；2. 已经成为Leader； 3. 整个列表中不存在在线的Leader，而节点处于在线节点列表的首位。 3. Beacon选举流程   Beacon选举基于节点IP地址标识，实现非常简单且高效。 "beacon":["192.168.1.11:16000", "192.168.1.12:16000"]   进程启动时首先检查Beacon集群配置，若未配置其他Beacon节点信息，则默认只有一个Beacon节点，此时该节点在启动时自动成为Leader节点。否则，向其他Beacon节点发送一个心跳消息，等待定时器回调检查并选举出Leader节点。选举流程如下图：

   checks whether it is online by checking whether heartbeat messages have been received from other Beacon nodes between timer callbacks. Check the traversal of m_mapBeacon to determine whether the node is online, and set the offline Leader node to offline state. If the current node should become a Leader node, it will become a Leader node.

4. Election communication between Beacon nodes

The election communication between the    Beacon nodes and the heartbeat of the nodes are integrated. The advantage of this is that when the leader node is not available, the fllower node can immediately become a leader node. In the election process, each fllower node only needs to traverse the heartbeat information of each Beacon node in its own memory, and there is no need to initiate an election when it is found that the leader is not online, which ensures the high availability of the cluster faster and better.

The heartbeat information of the    Beacon node carries the new data generated by the leader node as a coordination service. The fllower node completes the data synchronization while receiving the heartbeat, which ensures that when any fllower becomes a leader, it has obtained all the coordination information of the cluster and can be switched to leader at any time. In addition to the heartbeat triggered by the timer with the new data generated by the coordination service, the leader node immediately sends the heartbeat to the fllower when it generates the new data.

5. Realization of Beacon election

   Beacon heartbeat protocol proto:

/ * * @ brief BEACON Internode heartbeat * / message Election {int32 is_leader = 1; /

< 是否主节点 uint32 last_node_id = 2; ///< 上一个生成的节点ID repeated uint32 added_node_id = 3; ///< 新增已使用的节点ID repeated uint32 removed_node_id = 4; ///< 删除已废弃的节点ID}   检查Beacon配置，若只有一个Beacon节点则自动成为Leader： void SessionOnlineNodes::InitElection(const neb::CJsonObject& oBeacon){ neb::CJsonObject oBeaconList = oBeacon; for (int i = 0; i < oBeaconList.GetArraySize(); ++i) { m_mapBeacon.insert(std::make_pair(oBeaconList(i) + ".1", 0)); } if (m_mapBeacon.size() == 0) { m_bIsLeader = true; } else if (m_mapBeacon.size() == 1 && GetNodeIdentify() == m_mapBeacon.begin()->

First) {m_bIsLeader = true;} else {SendBeaconBeat ();}}

   sends Beacon heartbeat:

Void SessionOnlineNodes::SendBeaconBeat () {LOG4_TRACE ("); MsgBody oMsgBody; Election oElection; if (m_bIsLeader) {oElection.set_is_leader (1); oElection.set_last_node_id (m_unLastNodeId); for (auto it = m_setAddedNodeId.begin (); it! = m_setAddedNodeId.end ()) + + it) {oElection.add_added_node_id (* it);} for (auto it = m_setRemovedNodeId.begin (); it! = m_setRemovedNodeId.end (); + + it) {oElection.add_removed_node_id (* it);}} else {oElection.set_is_leader (0) } m_setAddedNodeId.clear (); m_setRemovedNodeId.clear (); oMsgBody.set_data (oElection.SerializeAsString ()); for (auto iter = m_mapBeacon.begin (); iter! = m_mapBeacon.end (); + + iter) {if (GetNodeIdentify ()! = iter- > first) {SendTo (iter- > first, neb::CMD_REQ_LEADER_ELECTION, GetSequence (), oMsgBody) }}}

   receives Beacon heartbeat:

Void SessionOnlineNodes::AddBeaconBeat (const std::string& strNodeIdentify, const Election& oElection) {if (! m_bIsLeader) {if (oElection.last_node_id () > 0) {m_unLastNodeId = oElection.last_node_id ();} for (int32 I = 0; I

< oElection.added_node_id_size(); ++i) { m_setNodeId.insert(oElection.added_node_id(i)); } for (int32 j = 0; j < oElection.removed_node_id_size(); ++j) { m_setNodeId.erase(m_setNodeId.find(oElection.removed_node_id(j))); } } auto iter = m_mapBeacon.find(strNodeIdentify); if (iter == m_mapBeacon.end()) { uint32 uiBeaconAttr = 1; if (oElection.is_leader() != 0) { uiBeaconAttr |= mc_uiLeader; } m_mapBeacon.insert(std::make_pair(strNodeIdentify, uiBeaconAttr)); } else { iter->

Second | = 1; if (oElection.is_leader ()! = 0) {iter- > second | = mc_uiLeader;}

   checks the online leader and becomes leader:

Void SessionOnlineNodes::CheckLeader () {LOG4_TRACE ("); std::string strLeader; for (auto iter = m_mapBeacon.begin (); iter! = m_mapBeacon.end (); + + iter) {if (mc_uiAlive & iter- > second) {if (mc_uiLeader & iter- > second) {strLeader = iter- > first } else if (strLeader.size () = = 0) {strLeader = iter- > first;}} else {iter- > second & = (~ mc_uiLeader);} uint32 uiLeaderBit = mc_uiLeader & iter- > second Iter- > second = ((iter- > second first = = GetNodeIdentify ()) {iter- > second | = 1;}} if (strLeader = = GetNodeIdentify ()) {m_bIsLeader = true;}} 6. Beacon node switches leader

   can easily view the status of Beacon nodes through nebcli, the command line management tool of Nebula cluster. For instructions on how to use nebcli, please see the README of the Nebcli project. Next, start three Beacon nodes, and repeatedly kill the Beacon process and restart to check the switching of the leader node.

   starts three beacon nodes:

Nebcli): show beaconnode is_leader is_online192.168.157.176:16000.1 yes yes192.168.157.176:17000.1 no yes192.168.157.176:18000.1 no yes

   kill drops the leader node:

Nebcli): show beaconnode is_leader is_online192.168.157.176:16000.1 no no192.168.157.176:17000.1 yes yes192.168.157.176:18000.1 no yes

   kill drops the fllower node:

Nebcli): show beaconnode is_leader is_online192.168.157.176:16000.1 no no192.168.157.176:17000.1 yes yes192.168.157.176:18000.1 no no

   restarts the two nodes dropped by kill:

Nebcli): show beaconnode is_leader is_online192.168.157.176:16000.1 no yes192.168.157.176:17000.1 yes yes192.168.157.176:18000.1 no yes

The    fllower node becomes a leader node after the original leader node is unavailable, and it will always be a leader node as long as there is no downtime, and will not switch again even if the original leader node becomes available again.

7. End

The purpose of    to develop Nebula framework is to provide a high-performance distributed service based on C++. If you think this article is useful to you, don't forget to go to Nebula's Github or Code Cloud to give a star, thank you.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.