How to install Kafka distributed Cluster in Linux

This article mainly introduces how to install Kafka distributed cluster in Linux. It is very detailed and has certain reference value. Friends who are interested must finish it!

Kafka (http://kafka.apache.org/) is a distributed messaging system written by LinkedIn using Scala, which is used as the basis for LinkedIn's activity flow (Activity Stream) and operational data processing pipeline (Pipeline), with high levels of scalability and high throughput. Both Spack and Elasticsearch support integration with Kafka. Here's a look at the comparison of several distributed open source message queuing systems:

Kafka cluster architecture:

Generally speaking, it is not recommended to directly use Zookeeper that comes with Kafka to set up zk clusters. Here we use independent zk clusters and zk clusters to build reference articles. ZooKeeper distributed cluster installation under Linux.

Here Linux chooses CentOS 7.2.

# cd / usr/local# mkdir kafka# cd kafka# wget-P / usr/local/kafka http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz# tar-zxvf kafka_2.11-1.0.0.tgz# mkdir kafka-logs# cd kafka_2.11-1.0.0/config# vim server.properties

Edit the configuration file (for configuration items, please refer to http://kafka.apache.org/documentation.html#brokerconfigs):

# unique ID. 0 starts the maximum size of broker.id=0# server port and ipport=9092host.name=192.168.2.20# log data saved directory log.dirs=/usr/local/kafka/kafka-logslog.retention.hours=168# for each message. Default 1000012message.max.bytes=5242880# default copy factor is the number of copies of saved messages, and default 1default.replication.factor=2# takes the maximum number of direct messages. Default 1048576replica.fetch.max.bytes=5242880# configuration zk address zookeeper.connect=192.168.20.11:2181192.168.20.12:2181192.168.20.13:2181

Before starting Kafka, you need to start Zookeeper, and then start Kafka, and each node needs to start separately:

# cd / usr/local/kafka/kafka_2.11-1.0.0/bin#. / kafka-server-start.sh-daemon.. / config/server.properties / / start the service in the background # jps / / check whether the service starts successfully #. / kafka-server-stop.sh / / shut down the service

Finally, we deploy a Kafka process on each of the three different servers to form a Kafka cluster.

These are all the contents of the article "how to install Kafka distributed clusters in Linux". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!