Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to build Kafka clusters". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let Xiaobian take you to learn "how to build Kafka clusters"!

I. Background

Recently, learning kafka, here is a record of the steps to build a kafka cluster on mac.

II. Installation of software

Because kafka depends on zookeeper, zookeeper needs to be installed, and kafka is written based on scala, which is based on jdk, so jdk needs to be installed.

JDK , recommended in jdk8 and above version, install jdk here slightly.

Zookeeper: Set up a 3-node zk pseudo-cluster locally.

Kafka builds a 3-node Kafka cluster locally.

3. Installation Step 1. Build a zk pseudo-cluster with 3 nodes

ip client connection port cluster election interface cluster atomic broadcast interface id node name in server.id, currently not used, this configuration into the hosts file

Note:

The id configuration in server.id is specified by creating the myid file in the directory specified by the dataDir configuration item in the zoo.cfg configuration file.

1. Download zookeeper installation package

get https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.6.2/apache-zookeeper-3.6.2-bin.tar.gz

2. Extract three copies and place them in the zookeeper directory

3. Modify the zookeeper configuration file, here zk01 is an example.

Change zoo_sample.cfg to zoo.cfg in conf directory

Edit the zoo.cfg configuration file

Note:

Here take zk01 as an example to write, because it is locally started on multiple, so you need to modify the port

The myid file values on each server should be different and unique.

The path of dataDir needs to be modified.

The rest of the configuration can be modified by looking at the specific situation.

Start 3 zk nodes

Enter the zk01/bin , zk02/bin , zk03/bin directories and execute the following commands

./ zkServer.sh --config ../ conf start

Connect zk cluster

./ zkCli.sh-server 127.0.0.1:2181,127.0.0.1:3181,127.0.0.1:41812, Build a kafka cluster with 3 nodes

Note:

The value of broker.id must be unique and numeric.

Download kafka

https://www.apache.org/dyn/closer.cgi? path=/kafka/2.6.0/kafka_2.13-2.6.0.tgz

2. Extract 3 copies and place them in the kafka directory

3. Modify the server.properties file, taking kafka01 as an example.

Properties to be modified. Take kafka01 as an example. Modify specific values for different interfaces.

Start three kafka nodes

Go to kafka01/bin , kafka02/bin , kafka03/bin directory in turn, and execute the following command

./ kafka-server-start.sh ../ config/server.properties &

5. kafka test

Take kafka01 as an example

1) Create a theme

bin/kafka-topics.sh --create --topic test-001 --replication-factor 1 --partitions 1 --bootstrap-server 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094

2. View subject information

bin/kafka-topics.sh --describe --bootstrap-server 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094 --topic test-001

Post a message to the topic you created

bin/kafka-console-producer.sh --topic test-001 --bootstrap-server 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094

4. Read the message just posted to the topic

bin/kafka-console-consumer.sh --topic test-001 --from-beginning --bootstrap-server 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094

At this point, a simple working kafka cluster is set up.

At this point, I believe everyone has a deeper understanding of "how Kafka clusters are built". Let's do it in practice! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.