Shulou(Shulou.com)06/03 Report--

1. Build Zookeeper Cluster

Zookeeper is a distributed open source framework that provides basic services for coordinating distributed applications. It exposes a set of common services to external applications-distributed synchronization, naming service, group maintenance, etc., simplifying the difficulty of distributed application coordination and management, and providing high-performance distributed services. ZooKeeper itself can be installed and run in standalone mode, but its strength lies in ensuring the stability and availability of ZooKeeper clusters based on certain policies through distributed ZooKeeper clusters (one Leader, multiple followers), thus achieving the reliability of distributed applications.

1. Download zookeeper-3.4.8.tar.gz at zookeeper. apache.org

2. Unzip tar -xzvf zookeeper-3.4.8.tar.gz

3. modify permissions sudo chown -R cms(ubuntu username) zookeeper-3.4.8

4. Modify the configuration file/etc/profile to add

5. Set parameters for Zookeeper's profile

Go to zookeeper-3.4.5/conf

1)cp zoo_sample.cfg zoo.cfg

2)Create a new directory under zookeeper to store data

mkdir zookerperdata

3)vim zoo.cfg

4)Note that master and slave1 are host names in the configuration shown above

The first port in the configuration file "server.id=host:port:port" above is the port number from the follower to the leader, and the second port is the port number for the leadership election.

5)Create myid

Next, create a file named myid under the directory specified by dataDir (zookeeper-3.4.8/zookerperdata/). The contents of the file have only one line, which corresponds to the id value of this host, that is, the id in server.id in the above figure. For example: myid in server 1 should be written 1.

vim myid

6)Remote copy to the same directory as slave1 and slave2

scp -r zookeeper-3.4.8 cms@slave1:/home/cms/

scp -r zookeeper-3.4.8 cms@slave1:/home/cms/

7)Change the myid values on slave1 and slave2 to 2 and 3 respectively

Start ZooKeeper Cluster

On each node of the ZooKeeper cluster, execute the script that starts the ZooKeeper service as follows:

QuorumPeerMain is the zookeeper process, which starts normally.

After starting Zookeeper on all machines in turn, you can view the startup status through Zookeeper scripts, including the role of each node in the cluster (or Leader or Follower), as shown below, which is the query result on each node in Zookeeper cluster.

Second, build a kafka cluster

1. download

Download official website: kafka.apache.org/downloads

Download version: the version corresponding to the version of Scala installed by yourself, personal habit is to download the previous version of the latest version

kafka_2.11-0.10.0.1.tgz

2. installation

tar -xzf kafka_2.11-0.10.0.1.tgz

cp kafka_2.11-0.10.0.1.tgz /home/cms/kafka

3. Configure environment variables

i.e. path, classpath, insignificant, optional

4. Modify the configuration file kafka/config/server.properties

5. Under kafka directory, create a directory where kafka stores data

mkdir kafkalogs

6. Other node configurations

Copy kafka files to other nodes

broker.id =1 #Unique id number in the whole cluster, integer, generally starting from 0

listeners=PLAINTEXT://192.168.31.132:9092 #protocol, current broker machine ip, port

port=9092 #broker port

host.name = 192.168.31.132 #broker machine ip

7. Start zookerper under each node

8. Start the kafka process in the kafka/bin directory of each node

--zookeeper : zookeeper cluster list, separated by commas. You don't need to specify the zookeeper node list in the whole cluster, but only specify a certain or several zookeeper node lists.

replication-factor: replication number, providing failover mechanism;1 means that there are data records on only one broker, and the value is generally greater than 1, which means that one piece of data will be automatically synchronized to other brokers to prevent data loss after a broker goes down.

partitions: A topic can be divided into multiple partitions, a consumer can consume multiple partitions, but a partition can only be consumed by a consumer, so increasing partitions can increase consumer throughput. Kafka only guarantees that messages within a partition are ordered, and data between multiple partitions is out of order.

9. Enabling producers and consumers

Producer: kafka-console-producer.sh --broker-list 192.168.31.131:9092 --topic test5

--broker-list : value can be one or more nodes in a broker cluster

Consumer:

kafka-console-consumer.sh --zookeeper 192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181 --topic test5 --from-beginning

--zookeeper : value can be one or more nodes in zookeeper cluster

--from-beginning means receiving from the first message

10. View topic

kafka-topics.sh --list --zookeeper 192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181

11. View topic details

kafka-topics.sh --describe --zookeeper 192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181 --topic test5

State description: test has three partitions 1, 2, 3, partition 0 leader is 3 (broker.id), partition 0 has three copies, and the state is lsr (ln-sync, indicating that you can participate in the election as leader).

12. Create topic with partition 3 and backup 3

bin/kafka-topics.sh --create --zookeeper 192.168.31.131:2181,192.168.31.132:2182,192.168.31.133:2183 --replication-factor 3 --partitions 3 --topic test5

13. Delete topic

Add delete.topic.enable=true to config/server.properties and restart the service.

kafka-topics.sh --delete --zookeeper 192.168.31.131:2181,192.168.31.132:2181,192.168.31.133:2181 --topic test5

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.