Shulou(Shulou.com)06/03 Report--

1 download and install JDK

Refer to the installation of jdk, jdk1.6+ is required.

2 installation of Kafka 2.1 download kafka

Download address: http://mirrors.hust.edu.cn/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz

2.2 extract the installation package

$tar zvxf kafka_2.11-0.10.0.1.tgz

2.3 copy the extracted folder to the installation directory

$cp-r kafka_2.11-0.10.0.1 / software/

2.4 create a soft link to the kafka installation directory

$ln-s / software/kafka_2.11-0.10.0.1/software/kafka

2.5 configure link variables

Modify the contents of / etc/profile file:

Export KAFKA_HOME=/software/kafka

Export PATH=$KAFKA_HOME/bin:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:

$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

Export CLASSPATH=.:$KAFKA_HOME/libs/*:$FLUME_HOME/lib/*:$HBASE_HOME/lib/*:$JAVA_

HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

2.6 start kafka

1) start kafka before starting zookeeper

$zookeeper-server-start.sh / software/kafka/config/zookeeper.properties &

2) start kafka

$kafka-server-start.sh / software/kafka/config/server.properties &

3 experience single node Kafka3.1 to create topic

Kafka-topics.sh-- create-- zookeeperlocalhost:2181-- replication-factor 1-- partitions 1-- topic test

Topic created by query

$kafka-topics.sh-list-zookeeperlocalhost:2181

3.2 send messages

$kafka-console-producer.sh-broker-listlocalhost:9092-topic test

This is a message

This is another message

3.3 consumption messages

$kafka-console-consumer.sh-bootstrap-serverlocalhost:9092-zookeeper localhost:2181-topic test-from-beginning

This is a message

This is another message

4 build a multi-broker Kafka cluster 4.1 replication configuration file

$cd / software/kafka/config/

$cp server.properties s1.properties

$cp server.properties s2.properties

$cp server.properties s3.properties

4.2 modify the configuration file

Config/s1.properties:

Broker.id=1

Listeners=PLAINTEXT://:9093

Log.dir=/tmp/kafka-logs-1

Config/s2.properties:

Broker.id=2

Listeners=PLAINTEXT://:9094

Log.dir=/tmp/kafka-logs-2

Config/s3.properties:

Broker.id=3

Listeners=PLAINTEXT://:9095

Log.dir=/tmp/kafka-logs-3

4.3 start kafka

$kafka-server-start.sh / software/kafka/config/s1.properties &

$kafka-server-start.sh / software/kafka/config/s2.properties &

$kafka-server-start.sh / software/kafka/config/s3.properties&

4.4 create topic

$kafka-topics.sh-create--zookeeper localhost:2181-replication-factor 3-partitions 1-topic my-replicated-topic

View topic description

$kafka-topics.sh-describe-zookeeper localhost:2181-topic my-replicated-topic

4.5 send messages

$kafka-console-producer.sh-broker-listlocalhost:9093-topic my-replicated-topic

My test msg 1

My test msg 2

My test msg 3

4.6 consumption message

$kafka-console-consumer.sh--zookeeper localhost:2181-from-beginning-topic my-replicated-topic

5 Fault tolerance Test 5.1 find S1 process

$ps-ef | grep s1.properties

5.2 Kill process S1

Kill pid

5.3 View topic description

$> kafka-topics.sh-describe-zookeeperlocalhost:2181-topic my-replicated-topic

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:

Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas:1,2,0 Isr: 2,0

5.4 launch consumer consumption theme message

$> kafka-console-consumer.sh--zookeeper localhost:2181-from-beginning-topic my-replicated-topic

My test message 1 my test message 26 fully distributed build 6.1 distribute kafka installation folders to the remaining cluster servers

$xsync kafka_2.11-0.10.0.1 /

$xsync kafka

6.2 distribute environment variables

# xsync / etc/profile

6.3 modify the server.properties properties file and distribute it

Broker.id=203 # modify the id of broker

Log.dirs=/home/hadoop/kafka/logs # the storage path of the modification log

Zookeeper.connect=s201:2181,s202:2181,s203:2181 # modify the connection path of zookeeper

6.4 start the Kafka cluster

Start the kafka cluster on each server that deploys kafka

$kafka-server-start.sh / software/kafka/config/server.properties &

$kafka-server-start.sh / software/kafka/config/server.properties &

$kafka-server-start.sh / software/kafka/config/server.properties &

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.