Shulou(Shulou.com)06/02 Report--

This section focuses on the deployment of kafka cluster environment

Introduction and strengthening of basic concepts of kafka

1) Producer: message producer, which is the client that sends messages to kafkabroker

2) Consumer: message consumer, the client that fetches messages from kafkabroker

3) Topic: can be understood as a queue

4) Consumer Group (CG): this is used by kafka to broadcast a topic message (sent to all consumer)

And unicast (sent to any consumer). A topic can have more than one CG. Topic messages will be replicated

(not really copied, but conceptually) to all CG, but each partion will only send a message to one of the CG

A consumer. If you need to implement broadcasting, as long as each consumer has a separate CG. To achieve

Unicast as long as all consumer are in the same CG. Consumer can also be grouped freely with CG without the need for

To send messages to different topic multiple times

5) Broker: a kafka server is a broker. A cluster consists of multiple broker. A broker

Can accommodate multiple topic

6) Partition: to achieve scalability, a very large topic can be distributed across multiple broker (that is, servers)

A topic can be divided into multiple partition, and each partition is an ordered queue. Every message in partition

Will be assigned an ordered id (offset). Kafka only guarantees that messages are sent in the order in one partition

Consumer, which does not guarantee the order of an topic as a whole (between multiple partition)

More partitions will increase the throughput of message processing to some extent, because kafka reads and writes based on files, so it also needs to open more file handles, which will also increase some performance overhead.

If there are too many partitions, then there will be a lot of log segments. When writing, it will become random because it is written in batches. Random Imax O will have a great impact on performance at this time. So generally speaking, Kafka can't have too much Partition.

7) replication-factor

The number of copies used to set the theme. Each topic can have multiple replicas, which are on different broker in the cluster, that is, the number of replicas cannot exceed the number of broker, otherwise the creation of the theme will fail.

When creating a Topic, there are two parameters that need to be filled in, namely partions and replication-factor. 8) the storage files of Offset:kafka are named after offset.kafka, and the advantage of using offset as a name is that it is easy to look up.

Looking for. For example, if you want to find the location at 2049, just find the 2048.kafka file. Of course, thefirstoffset will.

It's 00000000000.kafka.

Cluster environment build zookeeper cluster environment build download software package

Wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

Decompress

Create a configuration directory and modify files

[root@es7-1-80 conf] # cat zoo.cfg

InitLimit=10

SyncLimit=5

DataDir=/data/zk/data

ClientPort=2181

MaxClientCnxns=0

Server.1=172.16.0.80:2888:3888

Server.2=172.16.0.91:2888:3888

Server.3=172.16.0.92:2888:3888

Echo 1 > > = / data/zk/data/myid

Start, make standard startup service

Cat / etc/systemd/system/zookeeper.service

[Unit]

Description=zookeeper.service

After=network.target

[Service]

Type=forking

ExecStart=/usr/local/zookeeper/bin/zkServer.sh start

ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop

ExecReload=/usr/local/zookeeper/bin/zkServer.sh restart

[Install]

WantedBy=multi-user.target

Other nodes are configured as described above, and are started in turn

Cluster status View

/ usr/local/zookeeper/bin/zkServer.sh status

Kafka cluster build and download software package

Curl-LO https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz

Extract it to the desired location in

Create a configuration directory and modify files

Broker.id=0 #

The maximum number of threads for num.inetwork.threads=3 # broker to process messages, which generally does not need to be modified.

The number of threads num.io.threads=8 # # broker processes disk IO, which should be greater than the number of hard disks you have.

Socket.send.buffer.bytes=102400

Socket.receive.buffer.bytes=102400

Socket.request.max.bytes=104857600

Log.dirs=/usr/local/kafka/kafka_logs

Num.partitions=1 # the number of partitions per topic. If it is not specified when the topic is created, it will be overridden by the specified parameters when the topic is created.

Num.recovery.threads.per.data.dir=1

Offsets.topic.replication.factor=1

Transaction.state.log.replication.factor=1

Transaction.state.log.min.isr=1

Log.retention.hours=168

Log.segment.bytes=1073741824

Log.retention.check.interval.ms=300000

Zookeeper.connect=172.16.0.91:2181172.16.0.80:2181172.16.0.92:2181

Zookeeper.connection.timeout.ms=6000

Start, make standard startup service

[Unit]

Description=Apache Kafka server (broker)

After=network.target

After=syslog.target

After=zookeeper.target

[Service]

Type=forking

User=root

Group=root

ExecStart=/usr/local/kafka/bin/kafka-server-start.sh-daemon / usr/local/kafka/config/server.properties

ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

ExecReload=/bin/kill-HUP $MAINPID

KillMode=none

Restart=on-failure

RestartSec=5s

[Install]

WantedBy=multi-user.target

5. Add the kafka command to the environment variable

Export KAFKA_HOME=/usr/local/kafka

Export PATH=$PATH:$KAFKA_HOME/bin

Other nodes are deployed as described above, and broker.id is modified.

Configure Kafka public network IP address

Below kafka0.10

Modify server.properties configuration file

Advertised.host.name=xxxx

Advertised.port=9092

Above kafka0.10

Listeners=PLAINTEXT://0.0.0.0:9093 / / bind all ip

Advertised.listeners=PLAINTEXT://42.159.7.75:9093

Common operations of kafka

Create a theme

. / kafka-topics.sh-- zookeeper 172.16.0.80 create-- topic test1-- replication-factor 1-- partitions 3

When creating a Topic, there are two parameters that need to be filled in, namely partions and replication-factor.

Delete theme

. / kafka-topics.sh-- zookeeper 172.16.0.80 delete 2181-- topic test1

View a list of topics

. / kafka-topics.sh-- zookeeper 172.16.0.80 kafka-topics.sh 2181-- list

View topic status

Kafka-topics.sh-- zookeeper 172.16.0.80 describe 2181-- topic test2

Topic:test2 PartitionCount:3 ReplicationFactor:1 Configs:

Topic: test2 Partition: 0 Leader: 2 Replicas: 2 Isr: 2

Topic: test2 Partition: 1 Leader: 0 Replicas: 0 Isr: 0

Topic: test2 Partition: 2 Leader: 1 Replicas: 1 Isr: 1

Modify the theme partition

. / kafka-topics.sh-- zookeeper 172.16.0.80 create-- topic test3-- replication-factor 1-- partitions 1

Kafka-topics.sh-- zookeeper 172.16.0.80 partitions 2181-- alter-- topic test3-- partitions 3 kafka-topics.sh-- zookeeper 172.16.0.80 alter-- describe-- topic test3

Topic:test3 PartitionCount:3 ReplicationFactor:1 Configs:

Topic: test3 Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Topic: test3 Partition: 1 Leader: 2 Replicas: 2 Isr: 2

Topic: test3 Partition: 2 Leader: 0 Replicas: 0 Isr: 0

Production data

Kafka-console-producer.sh-- broker-list 172.16.0.80-- topic test4

Hi go

[2019-11-26 15 WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1: {test4=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2019-11-26 15 WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2: {test4=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[2019-11-26 15 WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3: {test4=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4: {test4=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClien)

Solution method

Port=9092

Change listeners=PLAINTEXT://172.16.0.92:9092 # to the corresponding ip

Re-production data

Kafka-console-producer.sh-- broker-list 172.16.0.80-- topic test4

Hello kafaka

Kafka-console-consumer.sh-- zookeeper 172.16.0.91-- topic test4-- from-beginning

Consumption data

Kafka-console-consumer.sh-- zookeeper 172.16.0.91-- topic test4-- from-beginning

List consumer main group

The consumer group, that is, Consumer Group, should be regarded as a design with bright spots in Kafka. So what is Consumer Group? To sum up in one sentence: Consumer Group is an extensible and fault-tolerant consumer mechanism provided by Kafka.

/ kafka-consumer-groups.sh-- bootstrap-server 172.16.0.80 list

Get the details of the new version of consumer group testgroup

/ kafka-consumer-groups.sh-- bootstrap-server 172.16.0.80 group testgroup-- describe

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.