Shulou(Shulou.com)05/31 Report--

How to achieve zookeeper technical analysis, many novices are not very clear about this, in order to help you solve this problem, the following editor will explain for you in detail, people with this need can come to learn, I hope you can gain something.

Zookeeper is a subproject of hadoop, although it comes from hadoop, but I find that zookeeper is more and more used to develop distributed frameworks outside the scope of hadoop. Today I want to talk about zookeeper, the following is not about how to use zookeeper, but what are the practical applications of zookeeper, which types of applications can give full play to the advantages of zookeeper, and finally talk about how zookeeper can play a role in distributed website architecture.

Zookeeper is a highly reliable coordination system for large-scale distributed systems. From this definition, we know that zookeeper is a coordination system and the object of action is a distributed system. Why do distributed systems need a coordination system? The reasons are as follows:

It is very difficult to develop a distributed system, which is mainly reflected in the "partial failure" of the distributed system. " "partial failure" means that when a message is transmitted between two nodes of the network, if the network fails, the sender cannot know whether the receiver has received the message, and the cause of the failure is very complicated. the receiver may or may not have received the message before the network error occurred, or the receiver's process may have died. The only way the sender can get the truth is to reconnect to the receiver and ask the receiver the cause of the error, which is the "partial failure" problem in distributed system development.

Zookeeper is the framework for solving the "partial failure" of distributed systems. Zookeeper does not let the distributed system avoid the problem of "partial failure", but enables the distributed system to deal with this kind of problem correctly when it encounters partial failure, so that the distributed system can run normally.

Next, I would like to talk about the practical application scenarios of zookeeper:

Scenario 1: there is a group of servers that provide certain services to the client (for example, the server of the distributed website I did earlier is a cluster of four servers to provide services to the front-end cluster). We hope that each time the client requests the server to find a server in the server cluster, so that the server can provide the client with the services needed by the client. For this scenario, there must be a list of this set of servers in our program, from which the list is read every time the client requests it. Then obviously this sub-list cannot be stored on a single-node server, otherwise the node will die and the whole cluster will fail. We hope that this list will be highly available. The highly available solution is that this list is distributed and is jointly managed by the server that stores the list. If a server in the storage list breaks down, other servers can immediately replace the broken server. And the broken server can be removed from the list, allowing the failed server to exit the operation of the entire cluster, and all these operations will not be operated by the failed server. It is done by a normal server in the cluster. This is an active distributed data structure, a data mechanism that can actively modify the state of data items when external conditions change. The Zookeeper framework provides this service. The name of this service is: unified naming Service, which is very similar to the JNDI service in javaEE.

Scenario 2: distributed lock service. When a distributed system operates data, such as reading data, analyzing data, and finally modifying data. In a distributed system, these operations may be distributed to different nodes in the cluster, so there is a problem of consistency in the process of data operation. if it is not consistent, we will get an incorrect operation result. in a single-process program, the problem of consistency is easy to solve, but it is more difficult in a distributed system. Because the operations of different servers in distributed systems are in independent processes, and the intermediate results and processes of operations have to be transmitted through the network, it is much more difficult to achieve data operation consistency. Zookeeper provides a lock service to solve this problem, which allows us to ensure the consistency of data operations when doing distributed data operations.

Scenario 3: configuration management. In a distributed system, we deploy a service application to n servers, and the configuration files of these servers are the same (for example, in the framework of the distributed website I designed, there are four servers on the server, the programs on the four servers are all the same, and the configuration files are all the same). If the configuration options of the configuration files change, then we have to change these configuration files one by one. If we need to change fewer servers, these operations are not too troublesome, if we have a large number of distributed servers, such as some large Internet companies' hadoop cluster has thousands of servers, then changing the configuration option is a troublesome and dangerous thing. At this point, zookeeper can come in handy. We can treat zookeeper as a highly available configuration memory and leave such things to zookeeper to manage. We copy the configuration files of the cluster to a node in the file system of zookeeper, and then use zookeeper to monitor the status of configuration files in all distributed systems. Once a configuration file is found to have changed, each server will be notified by zookeeper. Let each server synchronize the configuration files in the zookeeper, and the zookeeper service will also ensure that the synchronization operation is atomic, ensuring that the configuration files of each server can be updated correctly.

Scenario 4: provide fault repair for distributed systems. Cluster management is very difficult. Adding zookeeper service to the distributed system makes it easy for us to manage the cluster. The most troublesome thing in cluster management is node fault management. Zookeeper allows the cluster to select a healthy node as the master,master node to know the health of each server in the current cluster. Once a node fails, master will notify other servers in the cluster of the situation and reassign the computing tasks of different nodes. Zookeeper can not only find the fault, but also identify the faulty server to see what kind of fault the faulty server is. If the fault can be repaired, zookeeper can automatically fix it or tell the system administrator the cause of the error so that the administrator can quickly locate the problem and repair the node fault. You may also have a question, master malfunctioned, so what to do? Zookeeper also takes this into account. There is an "algorithm for electing leaders" within zookeeper, and master can choose dynamically. When master fails, zookeeper can immediately select a new master to manage the cluster.

Next, I would like to talk about the characteristics of zookeeper:

Zookeeper is a streamlined file system. It's a bit like hadoop, but zookeeper, the file system, manages small files, while hadoop manages very large files.

Zookeeper provides a wealth of "artifacts" that can implement many operations that coordinate data structures and protocols. Examples include distributed queues, distributed locks, and a "leader election" algorithm for a set of peer nodes.

Zookeeper is highly available, and its stability is quite good. Distributed clusters can rely on the management of zookeeper clusters, and zookeeper can be used to avoid single point of failure of distributed systems.

Zookeeper adopts a loosely coupled interaction mode. This is most evident in the distributed locks provided by zookeeper, where zookeeper can be used as a dating mechanism so that participating processes can discover and interact with each other without knowing about other processes (or networks), and the involved parties do not even have to exist at the same time, as long as a message is left in the zookeeper, and after the process ends, another process can read this information, thus decoupling the relationship between the nodes.

Zookeeper provides a shared repository for the cluster, from which the cluster can centrally read and write shared information, avoid the shared operation programming of each node, and reduce the difficulty of distributed system development.

The design of zookeeper adopts the design pattern of observer. Zookeeper is mainly responsible for storing and managing the data that everyone cares about, and then accepts the registration of observers. Once the state of these data changes, Zookeeper will be responsible for notifying those observers who have been registered on Zookeeper to respond accordingly, so as to achieve a similar Master/Slave management mode in the cluster.

Thus it can be seen that zookeeper is very conducive to distributed system development, it can make distributed systems more robust and efficient.

Not long ago, I joined the department's hadoop interest group. I installed hadoop, mapreduce, hive and hbase in the test environment. When installing hbase, I had to install zookeeper beforehand. At first, I installed zookeeper on all four servers, but my colleagues said that installing four and installing three is the same thing. This is because zookeeper requires that more than half of the machines are available before zookeeper can provide services, so more than half of the three are two. More than half of the four are also two, so installing three servers can achieve the effect of four servers. This problem shows that zookeeper usually chooses an odd number of servers when it is installed. In the process of learning hadoop, I feel that zookeeper is the most difficult sub-project to understand, not because it is technically responsible, but because its application direction makes me confused, so my first article on hadoop technology starts from zookeeper, and does not talk about specific technical implementation, but from the application scenario of zookeeper, understanding the field of zookeeper application, I want to learn zookeeper will get twice the result with half the effort.

The reason why I want to talk about zookeeper today is also a supplement to my last article on distributed website frameworks. Although I designed a distributed website architecture and made a simple fault handling mechanism, such as heartbeat mechanism, there is no way to deal with the single point of failure of the cluster. if a server goes down, the client will still try to connect to the server, resulting in blocking of some requests and waste of server resources. However, I do not want to modify my framework at present, because I always think that adding zookeeper services to existing services will affect the efficiency of the website. If there is a separate server cluster to deploy zookeeper, it is worth considering, but the server resources are too valuable, this is unlikely. Fortunately, our department has also found such a problem, our department will develop a strong remote invocation framework to separate cluster management and communication management, centrally provide efficient and available services, and so on. When the department's remote framework is developed, our website will add new services. I think our website will be more stable and efficient.

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.