Shulou(Shulou.com)06/01 Report--

This article mainly introduces the concepts that are easy to be confused in Kafka. It is very detailed and has a certain reference value. Friends who are interested must finish it!

Broker-list

Broker refers to the server side of kafka, which can be a server or a cluster. Both producer and consumer are equivalent to the client of this server.

Broker-list specifies one or more servers in the cluster. Generally, when we use console producer again, this parameter is a required parameter, and another required parameter is topic, as shown in the following example:

C:\ kafka\ kafka_2.12-1.1.1

λ.\ bin\ windows\ kafka-console-producer.bat-- broker-list localhost:9092-- topic test20190713

> this is a test

>

If the local host wants to simulate multiple broker, the method is to copy multiple server.properties, then modify the ports in it, and configure broker.id to simulate multiple broker clusters.

For example, in the case of simulating three broker, first make two copies of the server.properties in the config directory and name them server-1.properties and server-2.properties respectively, and then modify the following properties of the two files

Server-1.properties:

Broker.id=1

Listeners=PLAINTEXT://:9093

Log.dirs=C:/kafka/broker1

Server-2.properties:

Broker.id=2

Listeners=PLAINTEXT://:9094

Log.dirs=C:/kafka/broker2

Broker.id is used to uniquely identify each broker, and each broker has a unique id value to distinguish it from each other. When Kafka starts, it creates a virtual node with the name of the id of the current broker under the / brokers/ids path in zookeeper, and the health check of Kafka depends on this node.

We can open a client of zk and use the ls command to view the contents of this path:

λ.\ bin\ windows\ zookeeper-shell.bat localhost:2181

Connecting to localhost:2181

Welcome to ZooKeeper!

JLine support is disabled

Ls

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

Ls / brokers/ids

[0]

You can see the node with a broker.id of 0 that we started by default.

Bootstrap-servers vs zookeeper

Bootstrap-servers refers to the server address of the target cluster, which is the same as broker-list, except that we require the latter in console producer.

In the past, when we used console consumer to test sending and receiving messages, we would write:

C:\ kafka\ kafka_2.12-1.1.1

λ.\ bin\ windows\ kafka-console-consumer.bat-- zookeeper localhost:2181-- topic test20190713

Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].

In this way, you can receive the message sent by the producer console.

Now we can write the same way.

C:\ kafka\ kafka_2.12-1.1.1 λ.\ bin\ windows\ kafka-console-consumer.bat-- bootstrap-server localhost:9092-- topic test20190713

You can test it yourself, and you can also receive messages.

The former is the use of the old version, kafka before 0.8. the progress of consumption (offset) is written in zk, so consumer needs to know the address of zk. Later versions were all managed by broker, so they used bootstrap-server instead.

These are all the contents of this article entitled "what are the easily confused concepts in Kafka?" Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.