Shulou(Shulou.com)06/03 Report--

[TOC]

Verification of Kafka consumption form

The previous "Kafka Notes arrangement (1)" mentions the consumption form of consumers, which is explained as follows:

1. Each consumer belongs to a consumer group, and you can specify a group id. Group.id2, consumption form: within the group: consumers in the group consume the same data; at the same time, only one consumer can consume one partition; in one Topic and one consumer can consume messages in multiple partitions. Therefore, for a topic, it is recommended that you cannot consume more than the number of partitions in the same group at the same time, otherwise it will mean that some consumer will not get the message. Between groups: each consumer group consumes the same data and does not affect each other. 3. In the case of multiple threads in a consumer, one thread is equivalent to a consumer. For example, if the partition is 3, one consumer consumes three threads, and the other later consumer cannot consume.

Let's verify the consumption form of Kafka, but it is important to note that in the consumer's program code, the consumer's group.id can be specified (we will specify it in the configuration file below).

In fact, when using the shell command of kafka, you can also specify the configuration file to specify the consumer's group.id. If not, kafka will randomly generate a group.id (kafka.tools.ConsoleConsumer class in kafka-console-consumer.sh, if no group.id is specified, its policy is randomly generated).

In the later program code, we will use the same group.id to start four consumption threads (because the topic we created has three partition), and then open another consumer through kafka shell in the terminal to verify the kafka consumption form.

In addition, the topic used in the test is as follows:

$kafka-topics.sh-- describe hadoop-- zookeeper uplooking01:2181,uplooking02:2181,uplooking03:2181Topic:hadoop PartitionCount:3 ReplicationFactor:3 Configs: Topic:hadoop Partition: 0 Leader: 103 Replicas: 103101102 Isr: 103101102 Topic:hadoop Partition: 1 Leader: 101 Replicas: 101102103 Isr: 101102103 Topic:hadoop Partition: 2 Leader: 102103101 Isr: 102103101

That is, the partition is 3, and the copy is also 3.

The program code KafkaProducerOps.javapackage com.uplooking.bigdata.kafka.producer;import com.uplooking.bigdata.kafka.constants.Constants;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.Producer;import org.apache.kafka.clients.producer.ProducerRecord;import java.io.IOException;import java.io.InputStream;import java.util.Properties;import java.util.Random;/** * produces relevant data to Kafka topic through this KafkaProducerOps *

* Producer * / public class KafkaProducerOps {public static void main (String [] args) throws IOException {/ * specially load configuration file * configuration file format: * key=value * * minimize hard coding in the code * do not write the code to death, but configure * / Properties properties = new Properties () InputStream in = KafkaProducerOps.class.getClassLoader () .getResourceAsStream ("producer.properties"); properties.load (in) / * * two generic parameters * the first generic parameter: refers to the type of record key in kafka * the second generic parameter: refers to the type of record value in kafka * / String [] girls = new String [] {"Yao Huiying", "Liu Qian", "Zhou Xin", "Yang Liu"} Producer producer = new KafkaProducer (properties); Random random = new Random (); int start = 1; for (int I = start; I)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.