Shulou(Shulou.com)06/02 Report--

Basic introduction to kafka

The concept of a distributed stream processing platform, message subscription platform, generally has three features that allow you to publish and subscribe to streaming records. This is similar to message queuing or enterprise messaging systems. It can store flow records and has good fault tolerance. It can be processed as soon as the streaming record is generated.

Applicable scenario

A real-time streaming data pipeline is constructed, which can obtain data reliably between systems or applications. (equivalent to message queue)

Build real-time streaming applications to transform or influence these streaming data. (that is, stream processing, which changes internally between kafka stream topic and topic)

Deployment and installation

Kafka deployment installation depends on zoookper and java environments

Install the java environment

Yum install java-1.8.0-openjdk*-y

Install zoookper

Download the software package

Wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

Extract to a specified location

Modify the configuration file

Cp zoo_sample.cfg zoo.cfg

The configuration file is as follows to create the required directory

TickTime=2000

InitLimit=10

SyncLimit=5

DataDir=/usr/local/zookeeper_data

ClientPort=2181

Production standard service startup

Cat / etc/systemd/system/zookeeper.service

[Unit]

Description=zookeeper.service

After=network.target

[Service]

Type=forking

ExecStart=/usr/local/zookeeper/bin/zkServer.sh start

ExecStop=/usr/local/zookeeper/bin/zkServer.sh stop

ExecReload=/usr/local/zookeeper/bin/zkServer.sh restart

[Install]

WantedBy=multi-user.target

Systemctl start zookeeper

Install kafka

Curl-LO https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.1.0/kafka_2.12-2.1.0.tgz

Extract to a specified location

Modify the configuration file

Server.properties

Broker.id=1

Port=9092

Host.name=localhost

Log.dirs=/usr/local/kafka_logs

Zookeeper.connect=localhost:2181

Zookeeper.properties

DataDir=/usr/local/kafaka_data

ClientPort=2181

MaxClientCnxns=0

Host.name=localhost

Producer.properties

Metadata.broker.list=localhost:9092

Bootstrap.servers=localhost:9092

Compression.type=none

Consumer.properties

Bootstrap.servers=localhost:9092

Group.id=test-consumer-group

Zookeeper.connect=localhost:2181

* start up the standard service

Cat / etc/systemd/system/kafka.service

[Unit]

Description=Apache Kafka server (broker)

After=network.target

After=syslog.target

After=zookeeper.target

[Service]

Type=forking

User=root

Group=root

ExecStart=/usr/local/kafka/bin/kafka-server-start.sh-daemon / usr/local/kafka/config/server.properties

ExecStop=/usr/local/kafka/bin/kafka-server-stop.sh

ExecReload=/bin/kill-HUP $MAINPID

KillMode=none

Restart=on-failure

RestartSec=5s

[Install]

WantedBy=multi-user.target

Systemctl start kafka

test

Create a topic Topic

Bin/kafka-topics.sh-create-zookeeper localhost:2181-replication-factor 1-partitions 1-topic kafka01

Created topic "kafka01".

Bin/kafka-topics.sh-list-zookeeper localhost:2181

Start the message producer and send the message

Bin/kafka-console-producer.sh-broker-list localhost:9092-topic kafka01

Start the message consumer and receive the message

Bin/kafka-console-consumer.sh-zookeeper localhost:2181-topic kafka01-from-beginning

View topic list details

Bin/kafka-topics.sh-zookeeper localhost:2181-list

Bin/kafka-topics.sh-zookeeper localhost:2181-topic kafka01-describe

Topic:kafka01 PartitionCount:1 ReplicationFactor:1 Configs:

Topic: kafka01 Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Expand zoning

Bin/kafka-topics.sh-zookeeper localhost:2181-alter-topic kafka01-partitions 2

Bin/kafka-topics.sh-zookeeper localhost:2181-topic kafka01-describe

Topic:kafka01 PartitionCount:2 ReplicationFactor:1 Configs:

Topic: kafka01 Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Topic: kafka01 Partition: 1 Leader: 1 Replicas: 1 Isr: 1

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.