Shulou(Shulou.com)06/03 Report--

1. HA of kafka

(1) the role of zookeeper in kafka:

  -Zookeeper helps kafka clusters run: store some metadata, and also help kafka clusters manage (choose the master)

  -stores data about which topic consumers have consumed to that progress.

(2) problems solved by kafka HA:

Problem with   : in versions prior to 0.8, kafka does not provide high available mechanism. Once one or more broker goes down, the partition on it can no longer provide services during the downtime. If the broker never recovers or the disk fails, the data on it is lost. But for distributed, especially when the cluster rises to a certain scale, the possibility of one or more machines downtime is greatly increased, and the requirement for failover is very high. As a result, kafka began to provide highly available mechanisms after version 0.8.

The problem solved by   : the HA of Kafka is at the data level, that is, if a piece of data is lost, Consumer can still be consumed normally (it also means that a broker is down and the data above will not be lost).

   solution: use replication (copy, data backup of the same data). Among them, there are master replicas and slave replicas (selected by zookeeper). Each broker will store some master replicas to ensure the load balance of the kafka cluster.

(3) the process of writing data by kafka HA:

  -use zookeeper to select a leader for multiple copies of the broker

The leader of     is equivalent to a temporary node in zookeeper. If the leader goes down, the zookeeper will select the master again. If all the follower in the ISR list goes down, it will wait until a follower in the ISR list can be used again. This will turn the follower into a leader. (ISR is a list of available cluster nodes maintained by kafka in zookeeper.)

  -when writing data, first write data to leader (master copy)

  -other broker (follower) to fetch data in leader (from replica)

  -after data fetching is completed, the information is returned to leader (ack)

Explanation: an ISR (list of synchronized copies) is maintained in leader, and all valid follwer are stored in the list. If the data of follower and leader are too far behind (default is 4000 pieces of data) or if no fetch request is sent to leader for a certain period of time (default is 10 seconds), the follwer will be automatically removed from the ISR list and in the ISR list. Each follower fetch completion data, will send data to the leader, and eventually if the leader receives all the follower feedback, and then to the producer feedback, send ACK.

Where ACK represents the return level of data written by Producer:

0 means that as long as the producer sends data, it means that the data is written successfully. 1: as long as the leader is successfully written, it means that the data is written successfully. All /-1 means that all copies are written successfully before data is written successfully.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.