Shulou(Shulou.com)06/03 Report--

Editor to share with you how to deploy and install kafka under Centos6.5. I hope you will get something after reading this article. Let's discuss it together.

I) install JDK

Step 1: create a new path / usr/Java and extract the jdk-7u71-linux-x64.tar.gz under it

# mkdir-p / usr/java

# tar-zxvf jdk-7u71-linux-x64.tar.gz

Step 2: add JDK to the system environment variable

# vi / etc/profile

Add the following:

Export JAVA_HOME=/usr/java/jdk1.7.0_71

Export PATH=$JAVA_HOME/bin:$PATH

Export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

Step 3: make the configuration effective

Source / etc/profile # makes the configuration file effective immediately

# java-version / / JDK version java version "1.7.0x79" is also fine, which is not supported by the current JDK1.8 version

Java version "1.7.0Y71"

Java (TM) SE Runtime Environment (build 1.7.0_71-b14)

Java HotSpot (TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

II) install zookeeper

Step 1: extract the zookeeper installation package (path independent, installation path can be customized)

# tar-zxvf zookeeper-3.3.6.tar.gz

Step 2: configure environment variables

# vim / etc/profile

Export zookeeper_home=/home/laoyang/zookeeper

# source / etc/profile

# sh $zookeeper_home/bin/zkServer.sh start

JMX enabled by default

Using config: / home/laoyang/zookeeper/bin/../conf/zoo.cfg

Grep: / home/laoyang/zookeeper/bin/../conf/zoo.cfg: No such file or directory

Starting zookeeper... STARTED

III) install kafka

Step 1: extract & move to / usr/local

# tar-xzvf kafka_2.11-0.9.0.1.tgz

# mv kafka_2.11-0.9.0.1 / usr/local/kafka

Step 2: configure kafka and modify server.properties

# mkdir / usr/local/kafka/log/kafka # create a kafka log directory

# cd / usr/local/kafka/config # enter the configuration directory

# vim server.properties # Edit and modify the corresponding parameters

Broker.id=0

Port=9092 # Port number

Host.name=10.8.5.101 # server IP address, modified to your own server IP

Log.dirs=/usr/local/kafka/log/kafka # log storage path, the directory created above

Zookeeper.connect=localhost:2181 # zookeeper address and port, standalone configuration deployment, localhost:2181

Step 3. Configure zookeeper under kafka

# mkdir / usr/local/kafka/zookeeper # create a zookeeper directory

# mkdir / usr/local/kafka/log/zookeeper # create a zookeeper log directory

# cd / usr/local/kafka/config # enter the configuration directory

# vim zookeeper.properties # Edit and modify the corresponding parameters

DataDir=/usr/local/kafka/zookeeper # zookeeper data directory

DataLogDir=/usr/local/kafka/log/zookeeper # zookeeper log directory

ClientPort=2181

MaxClientCnxns=100

TickTime=2000

InitLimit=10

SyncLimit=5

Step 4: create a startup and shutdown kafka script

1) create a startup kafka script

# cd / usr/local/kafka/bin/

# vim kafkastart.sh

#! / bin/bash

# start zookeeper

/ usr/local/kafka/bin/zookeeper-server-start.sh / usr/local/kafka/config/zookeeper.properties &

Sleep 3

# Start kafka

/ usr/local/kafka/bin/kafka-server-start.sh / usr/local/kafka/config/server.properties &

2) create a shutdown kafka script

# vim kafkastop.sh

#! / bin/bash

# stop zookeeper

/ usr/local/kafka/bin/zookeeper-server-stop.sh / usr/local/kafka/config/zookeeper.properties &

Sleep 3

# Stop kafka

/ usr/local/kafka/bin/kafka-server-stop.sh / usr/local/kafka/config/server.properties &

3) # add script execution permissions

Chmod + x kafkastart.sh

Chmod + x kafkastop.sh

4) set the script to be automatically executed on boot

# vim / etc/rc.d/rc.local # Edit, add a line at the end

Sh / usr/local/kafka/kafkastart.sh & # set boot to run scripts in the background automatically

Sh / usr/local/kafka/kafkastart.sh # launch kafka

Sh / usr/local/kafka/kafkastop.sh # turn off kafka

So far, the installation and configuration of Kafka under CentOs6.5 is complete.

Sign of success:

IV) Test kafka

Step 1: start zookeeper

# sh $zookeeper_home/bin/zkServer.sh start

JMX enabled by default

Using config: / home/laoyang/zookeeper/bin/../conf/zoo.cfg

Grep: / home/laoyang/zookeeper/bin/../conf/zoo.cfg: No such file or directory

Starting zookeeper... STARTED

Step 2: start kafka

#. / kafkastart.sh

Step 3: test create topic

# cd / usr/local/kafka/bin

#. / kafka-topics.sh-create-zookeeper localhost:2181-replication-factor 1-partitions 1-topic test

Step 4: view the created topic through the list command

# cd / usr/local/kafka/bin

#. / kafka-topics.sh-list-zookeeper localhost:2181

Step 5: production message test

#. / kafka-console-producer.sh-broker-list localhost:9092-topic test

Kafka test! / / the "Kafka test!" here For manual input, "ctrl + c" exits after input

Step 6: consume the message test

#. / kafka-console-consumer.sh-zookeeper localhost:2181-topic test-from-beginning

Isimm laoyang # previously tested the input

Kafka test!

After the above steps, the kafka installation is successful.

V) stop kafka

Step 1: stop Kafka

Cd / usr/local/kafka

#. / kafkastop.sh / / the kafka process here has not been turned off, and manual kill is required

Step 2: stop Zookeeper server

# sh $zookeeper_home/bin/zkServer.sh stop

At this point, start, test, and stop the whole process of ok.

Error during kafka startup:

WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)

EndOfStreamException: Unable to read additional data from client sessionid 0x157ac94cd940002, likely client has closed socket

At org.apache.zookeeper.server.NIOServerCnxn.doIO (NIOServerCnxn.java:228)

At org.apache.zookeeper.server.NIOServerCnxnFactory.run (NIOServerCnxnFactory.java:208)

At java.lang.Thread.run (Thread.java:745)

Reason for reporting an error:

Caused by the inconsistency of jar versions between the zookeeper server and the client

Solution:

Just copy zookeeper-xxx.jar from the root of the zookeeper server installation directory to the libs directory under the kafka installation directory, and delete the jar package under the libs directory under kafka.

Error report:

Java.lang.RuntimeException: A broker is already registered on the path / brokers/ids/1. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.

At kafka.utils.ZkUtils$.registerBrokerInZk (ZkUtils.scala:205)

At kafka.server.KafkaHealthcheck.register (KafkaHealthcheck.scala:57)

At kafka.server.KafkaHealthcheck.startup (KafkaHealthcheck.scala:44)

At kafka.server.KafkaServer.startup (KafkaServer.scala:103)

At kafka.server.KafkaServerStartable.startup (KafkaServerStartable.scala:34)

At kafka.Kafka$.main (Kafka.scala:46)

At kafka.Kafka.main (Kafka.scala)

Reason for reporting an error:

The error is due to a duplicate broker.id in server.properties

Solution:

# vim server.properties

# The id of the broker. This must be set to a unique integer for each broker.

Broker.id=0

After reading this article, I believe you have a certain understanding of "how to deploy and install kafka under Centos6.5". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.