Shulou(Shulou.com)06/01 Report--

Today, the editor will share with you the relevant knowledge points about how to deploy the distributed messaging system Kafka under Linux. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article.

Kafka, originally developed by Linkedin, is a distributed, partitioned (partition), multi-replica (replica), distributed messaging system based on zookeeper coordination. Its biggest feature is that it can process a large amount of data in real time to meet a variety of demand scenarios: such as hadoop-based batch processing system, low-latency real-time system, storm/Spark streaming engine, web/nginx log, access log, message service and so on.

Kafka:2.12-2.5.0

1. Install JDK 1.8yum-y install java-1.8.0*

The java-version command checks the JDK version, as shown in the figure. The installation is successful:

2. Install the zookeeper service

Reference article: installation and use of ZooKeeper under Linux

3. Create a working directory Download installation package: # create installation directory mkdir-p / opt/kafka# move to directory cd / opt/kafka# download kafka installation package wget https://mirrors.aliyun.com/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz# extract tar-zxvf kafka_2.12-2.5.0.tgz4, Configuration file: # enter the configuration directory cd kafka_2.12-2.5.0/config/# backup configuration file cp server.properties server.properties.bak# modify configuration file vim server.properties# modify and add the following configuration broker.id=1listeners=PLAINTEXT://127.0.0.1:9092advertised.listeners=PLAINTEXT://127.0.0.1:9092# other custom configuration (based on actual modification) zookeeper.connect=127.0.0.1:2181zookeeper .connection.timeout.ms = 1800 save exit: wq

# configuration description broker.id: the unique ID of the current machine in the cluster. For example, if there are three Kafka hosts, they will be configured as 1pm, 2pm and 3pm respectively.

Listeners: service listening port.

Advertised.listeners: the port number provided to the producer and consumer, that is, the external access address. Default is the value of listeners.

Zookeeper.connect:zookeeper connection address. If there is a cluster configuration, each Kafka CVM needs to connect to all zookeeper services. Examples are as follows:

Zookeeper.connect=192.168.1.41:2181192.168.1.42:2181192.168.1.47:2181

Zookeeper.connection.timeout.ms:zookeeper connection timeout.

5. Start Kafka:

(first make sure zookeeper is started)

# move to the working directory cd / opt/kafka/kafka_2.12-2.5.0/bin/# to start kafka./kafka-server-start.sh-daemon.. / config/server.properties# to shut down the kafka service. / kafka-server-stop.sh

Check that the port has been monitored and started successfully:

6 、 The test creates a topic:# and moves it to the working directory cd / opt/kafka/kafka_2.12-2.5.0/bin/# to create topic./kafka-topics.sh-- create-- zookeeper 127.0.0.1opt/kafka/kafka_2.12 2181-- replication-factor 1-- partitions 1-- topic topic1# to view topic information. / kafka-topics.sh-- describe-- zookeeper 127.0.0.1topic:# 2181-- topic topic1# starts the producer console. / kafka- Console-producer.sh-- broker-list 127.0.0.1 kafka-console-consumer.sh-- topic topic1# launches the consumer console (a new window opens). / kafka-console-consumer.sh-- bootstrap-server 127.0.0.1 kafka-console-consumer.sh-- topic topic1-- from-beginning

At this point, a test message is sent on the generator console, and the consumer console will receive:

These are all the contents of the article "how to deploy distributed messaging system Kafka under Linux". Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.