Shulou(Shulou.com)06/02 Report--

This article will explain in detail how to install kafka in centos, the content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Introduction to 1.kafka technical terms

Broker kafk instance. The kafka cluster contains one or more instances. In principle, there is only one instance on a host.

Topic kafka message topic, the message of kafka must belong to a topic, and there are multiple Topic in a Broker.

Partition

It is the concept of the physical layer, called partitions, and each Topic contains one or more partitions.

Producer

Message publisher, that is, the Broker responsible for publishing messages to kafka

Consumer

Message consumer, that is, the client responsible for consuming messages to kafka Broker

Consumer Group

Each consumer must belong to a consumer group

1. Download kafka

$cd / usr/java

[root@43-c59438365-0048-0727982 java] # cd kafka

[root@43-c59438365-0048-0727982 kafka] # wget http://mirror.bit.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz

Thus, the version of kafka I downloaded is 2.3.0.

two。 Decompress kafka

[root@43-c59438365-0048-0727982 kafka] # ll

Total 55936

-rw-r--r-- 1 root root 57215197 Jun 25 08:38 kafka_2.12-2.3.0.tgz

[root@43-c59438365-0048-0727982 kafka] # tar zxvf kafka_2.12-2.3.0.tgz

...

[root@43-c59438365-0048-0727982 kafka] # ll

Total 55940

Drwxr-xr-x 6 root root 4096 Jun 20 04:44 kafka_2.12-2.3.0

-rw-r--r-- 1 root root 57215197 Jun 25 08:38 kafka_2.12-2.3.0.tgz

[root@43-c59438365-0048-0727982 kafka] #

3. Configure kafka environment variables

$vi / etc/profile

Add the following two lines

Export KAFKA_HOME=/usr/java/kafka/kafka_2.12-2.3.0

Export PATH=$PATH:$KAFKA_HOME/bin

After exiting, execute the refresh command

Source profile

4. Start kafka

Because kafka depends on zookeeper, you need to start zookeeper before starting kafka. So we need to find the configuration file for zookeeper, as follows

[root@43-c59438365-0048-0727982 config] # ll

Total 68

-rw-r--r-- 1 root root 906 Jun 20 04:43 connect-console-sink.properties

-rw-r--r-- 1 root root 909 Jun 20 04:43 connect-console-source.properties

-rw-r--r-- 1 root root 5321 Jun 20 04:43 connect-distributed.properties

-rw-r--r-- 1 root root 883 Jun 20 04:43 connect-file-sink.properties

-rw-r--r-- 1 root root 881 Jun 20 04:43 connect-file-source.properties

-rw-r--r-- 1 root root 1552 Jun 20 04:43 connect-log4j.properties

-rw-r--r-- 1 root root 2262 Jun 20 04:43 connect-standalone.properties

-rw-r--r-- 1 root root 1221 Jun 20 04:43 consumer.properties

-rw-r--r-- 1 root root 4727 Jun 20 04:43 log4j.properties

-rw-r--r-- 1 root root 1925 Jun 20 04:43 producer.properties

-rw-r--r-- 1 root root 6851 Jun 20 04:43 server.properties

-rw-r--r-- 1 root root 1032 Jun 20 04:43 tools-log4j.properties

-rw-r--r-- 1 root root 1169 Jun 20 04:43 trogdor.conf

-rw-r--r-- 1 root root 1023 Jun 20 04:43 zookeeper.properties

[root@43-c59438365-0048-0727982 config] # pwd

/ usr/java/kafka/kafka_2.12-2.3.0/config

[root@43-c59438365-0048-0727982 config] #

As above is the absolute path to the zookeeper configuration file, then start zookeeper, go to the kafka installation directory bin, and execute the following startup command

[root@43-c59438365-0048-0727982 bin] #. / zookeeper-server-start.sh / usr/java/kafka/kafka_2.12-2.3.0/config/zookeeper.properties &

[1] 25053

[root@43-c59438365-0048-0727982 bin] # [2019-08-24 15 INFO Reading configuration from: / usr/java/kafka/kafka_2.12-2.3.0/config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)

...

[2019-08-24 15 31lo 29407] INFO tickTime set to (org.apache.zookeeper.server.ZooKeeperServer)

[2019-08-24 15 31 29407] org.apache.zookeeper.server.ZooKeeperServer-1

[2019-08-24 15 31 29407] org.apache.zookeeper.server.ZooKeeperServer-1

[2019-08-24 15 31 29413] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)

[2019-08-24 15 31VR 29416] INFO binding to port 0.0.0.0Unix 0.0.0.0pur2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

^ C

[root@43-c59438365-0048-0727982 bin] #

As shown above, zookeeper started successfully.

Then start the kafka service

Kafka service startup requires a kafka configuration file, which is located in the same location as the zookeeper configuration file, such as / usr/java/kafka/kafka_2.12-2.3.0/config

[root@43-c59438365-0048-0727982 bin] #. / kafka-server-start.sh / usr/java/kafka/kafka_2.12-2.3.0/config/server.properties &

[2] 25406

[root@43-c59438365-0048-0727982 bin] # [2019-08-24 15 purge 37purl 12829] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)

[2019-08-24 15 HUP 3714] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)

[2019-08-24 15 3714 1490] INFO starting (kafka.server.KafkaServer)

[2019-08-24 15 3714 1491] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)

...

[2019-08-24 15 Started data-plane processors for 3714 823] INFO [SocketServer brokerId=0] 1 acceptors (kafka.network.SocketServer)

[2019-08-24 15 3714 830] INFO Kafka version: 2.3.0 (org.apache.kafka.common.utils.AppInfoParser)

[2019-08-24 15 3714 830] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser)

INFO Kafka startTimeMs: 1566632234826 (org.apache.kafka.common.utils.AppInfoParser)

[2019-08-24 15 started 3714 831] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Thus it can be seen that kafka started successfully.

5. Test kafka

Create Topic

[root@43-c59438365-0048-0727982 bin] # kafka-topics.sh-create-zookeeper localhost:2181-topic myTopic-partitions 1-replication-factor 1

[2019-08-24 16-15-15-1854] INFO Accepted socket connection from / 0-0-8-24 16-15-15-18-54 / 0-0-8-24 16-15-15-18-54 / 0-08-24 16-15-15-18-54 [008-24 16-15-15-1854]

[2019-08-24 16-15-15-1857] INFO Client attempting to establish new session at / 0-0-8-24 16-15-15-18-57 / 0-0-8-24 16-15-15-18-57 [008-24 16-15-15-1857] 016-08-24 16-15-15-18-57]

[2019-08-24 16-15-18-60] INFO Established session 0x10085c3259c0002 with negotiated timeout 30000 for client / 0-0 for client 0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0-0, 0-0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 0, 0, 0, 0, 0, 0, 0, 0, 4, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 4, 8, 0, 0, 0, 0, 4, 4, 8, 0, 0, 0, 0, 0, 0, 0, 4, 4, 8, 0, 0, 0, 0, 0, 4, 8, 8, 0, 0, 0, 0, 0, 0, 0,

[2019-08-24 16 config/topics/myTopic 15 config/topics/myTopic 22126] INFO Got user-level KeeperException when processing sessionid:0x10085c3259c0002 type:setData cxid:0x4 zxid:0x44 txntype:-1 reqpath:n/an Error Path:/config/topics/myTopic Error:KeeperErrorCode = NoNode for / config/topics/myTopic (org.apache.zookeeper.server.PrepRequestProcessor)

Created topic myTopic.

As you can see from the above, we created a theme called myTopic.

Create producer [root@43-c59438365-0048-0727982 bin] # kafka-console-producer.sh-- topic myTopic-- broker-list localhost:9092

$>

It can be seen from the above that the producer has created it.

Create consumers

It's obvious here that a new window is needed.

[root@43-c59438365-0048-0727982 bin] # kafka-console-consumer.sh-topic myTopic-bootstrap-server localhost:9092

As follows, you can send a message in the producer window, and the consumer window receives the message in time.

# # opening the public network of 6.kafka

Set the server.properties file for kafka

Private network ip:9092 of listeners=PLAINTEXT:// server

Advertised.listeners=PLAINTEXT:// server public network ip:9092

View the private network ip of the server through ifconfig. View the server's public network ip through the mapping relationship between the private and public networks.

After running for a period of time, it is found that kafka has been killed. Starting the kafka service with the following command can avoid this problem.

. / kafka-server-start.sh-daemon / usr/java/kafka/kafka_2.12-2.5.0/config/server.properties &

Add-daemon after starting the command

It's over ever since.

On how to install kafka in centos to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.