Shulou(Shulou.com)06/01 Report--

This article is to share with you about the installation and deployment of single-node KAFKA, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Project background:

The project needs to use incremental data synchronization tools to access big data platform to achieve incremental synchronization function. According to the company's requirements, the kafka version is 0.11.

In this paper, we build a single node kafka.

Download address

Http://kafka.apache.org/downloads.html

First, test environment information (personal virtual machine)

[root@zrd soft] # uname-a

Linux zrd 2.6.32-696.el6.x86_64 # 1 SMP Tue Mar 21 19:29:05 UTC 2017 x86 "64 GNU/Linux

[root@zrd soft] # free-h

Total used free shared buffers cached

Mem: 1.8G 448M 1.4G 1.1M 26M 204M

-/ + buffers/cache: 216M 1.6G

Swap: 1.0G 0B 1.0G

[root@zrd soft] # df-h

Filesystem Size Used Avail Use% Mounted on

/ dev/sda2 29G 11G 18G 39% /

Tmpfs 932M 72K 932m 1% / dev/shm

/ dev/mapper/vg_test-lv_test

21G 211m 20G 2% / data

/ CentOS-6.9-x86_64-LiveDVD.iso

1.9g 1.9g / yumdir

[root@zrd soft] # java-version

Java version "1.8.0,144"

Java (TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot (TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

II. Brief introduction of kafka

Kafka is a high-throughput distributed publish and subscribe messaging system that can handle all action flow data in consumer-scale websites. This action (web browsing, search and other user actions) is a key factor in many social functions on the modern web. This data is usually resolved by processing logs and log aggregations due to throughput requirements.

Introduction to 2.1kafka catalogue

/ bin executable script for operating kafka, including scripts under windows

/ config configuration file directory

/ libs dependent library directory

/ logs log data directory, directory kafka divides server logs into five types, including: server,request,state,log-cleaner,controller

Interpretation of 2.2kafka nouns

Producer: producer.

Consumer: consumers.

Topic: messages are recorded with topic as the category, and Kafka classifies the message Feed into categories, and each type of message is called a Topic.

Broker: runs as a cluster and can consist of one or more services, each called a broker;. Consumers can subscribe to one or more topics (topic) and pull data from Broker to consume these published messages.

Each message (also known as a record record, also known as a message) consists of a key, a value, and a timestamp.

2.3kafka has four core API introductions

The application uses producer API to publish messages to one or more topic.

The application uses consumer API to subscribe to one or more topic and process the resulting messages.

The application uses streams API as a stream processor, consuming the input stream from one or more topic and generating an output stream to one or more topic, effectively switching the input stream to the output stream.

Connector API allows you to build or run reusable producers or consumers to link topic to existing applications or data systems.

2.4kafka application scenario

Build a real-time streaming data pipeline to reliably obtain the data between the system and the application.

Build real-time streaming applications to transform or react to data streams.

2.5kafka Consumer Group

What is consumer group? In a word, consumer group is a scalable and fault-tolerant consumer mechanism provided by kafka. Since it is a group, there must be multiple consumers or consumer instances (consumer instance) within the group that share a common ID, the group ID. All consumers in the group coordinate to consume all the partition of the subscription topic (subscribed topics). Of course, each partition can only be consumed by one consumer within the same consumer group.

Three features:

There can be one or more consumer instance,consumer instance under 1consumer group, which can be a process or a thread.

2group.id is a string that uniquely identifies a consumer group

Each partition under the subscribed topic under 3consumer group can only be assigned to one consumer under a group (of course, this partition can also be assigned to other group)

Third, installation steps

3.1 install jdk (install by yourself)

3.2 install zookeeper (single node)

[root@zrd soft] # tar-zxvf zookeeper-3.4.10.tar.gz

Create two directories, data and logs, under the home directory to store data and logs

Create a new zoo.cfg file in the conf directory and save it by writing the following

TickTime=2000

DataDir=/soft/zookeeper-3.4.10/data

DataLogDir=/soft/zookeeper-3.4.10/logs

ClientPort=2181

Go to the bin directory, start, stop, restart points, and view the current node status (including the roles in the cluster). Do not execute:

. / zkServer.sh start

. / zkServer.sh stop

. / zkServer.sh restart

. / zkServer.sh status

Touch / soft/zookeeper-3.4.10/data/myid

Echo'1' > data/myid

3.2 install kafka

3.2.1 extract the package

Tar-zxvf kafka_2.11-0.11.0.2

3.2.2 configuration

[root@zrd config] # ls

Connect-console-sink.properties connect-log4j.properties server.properties

Connect-console-source.properties connect-standalone.properties tools-log4j.properties

Connect-distributed.properties consumer.properties zookeeper.properties

Connect-file-sink.properties log4j.properties

Connect-file-source.properties producer.properties

[root@zrd config] # pwd

/ soft/kafka_2.11-0.11.0.2/config

/ / the three most important configurations of kafka are: broker.id, log.dir, and config/server.properties parameters on zookeeper.connect,kafka server are described and explained as follows: https://blog.csdn.net/lizhitao/article/details/25667831

[root@zrd config] # vim server.properties

Broker.id=0

Delete.topic.enable=true

Listeners=PLAINTEXT://192.168.15.131:9092

Num.network.threads=3

Num.io.threads=8

Socket.send.buffer.bytes=102400

Socket.receive.buffer.bytes=102400

Socket.request.max.bytes=104857600

Log.dirs=/data/kafka/data

Num.partitions=1

Num.recovery.threads.per.data.dir=1

Offsets.topic.replication.factor=1

Transaction.state.log.replication.factor=1

Transaction.state.log.min.isr=1

Log.flush.interval.messages=10000

Log.flush.interval.ms=1000

Log.retention.hours=168

Log.retention.bytes=1073741824

Log.segment.bytes=1073741824

Log.retention.check.interval.ms=300000

Zookeeper.connect=192.168.15.131:2181192.168.15.132:2181192.168.15.133:2181

Zookeeper.connection.timeout.ms=6000

Group.initial.rebalance.delay.ms=0

Start the process

. / kafka-server-start.sh. / kafka/config/server.properties

Test: create topic-- test

[root@zrd bin] # jps

4729 Kafka

3049 QuorumPeerMain

5019 Jps

4238 QuorumPeerMain

[root@zrd bin] #. / kafka-topics.sh-- create-- zookeeper 192.168.6.105 create-- replication-factor 1-- partitions 1-- topic test

Created topic "test".

Another window: list the topic that has been created

[root@zrd bin] #. / kafka-topics.sh-- list-- zookeeper 192.168.6.105purl 2181

Test

Impersonate the client to send messages

[root@zrd bin] #. / kafka-console-producer.sh-- broker-list 192.168.6.105 topic r7_test

> hello world

[2018-05-17 15 org.apache.kafka.clients.NetworkClient 2614] WARN Error while fetching metadata with correlation id 1: {r7_test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

> 20180517

Impersonate the client to receive messages

[root@zrd bin] #. / kafka-console-consumer.sh-- zookeeper 192.168.6.105 zookeeper 2181-- from-beginning-- topic r7_test

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

The above is how the installation and deployment of single-node KAFKA is, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.