Shulou(Shulou.com)06/01 Report--

This article mainly introduces what the programming model of kafka has, which can be used for reference by interested friends. I hope you can learn a lot after reading this article.

1.kafka consumer programming model

Regional consumption model

Group consumption model

1.1. Regional consumption model

1.1.1. Partition consumption architecture diagram, with each partition corresponding to a consumer.

1.1.2. Pseudo-code description of partition consumption model

Specifies the offset, which is used to start consumption from the place where it was last consumed.

Submit the offset, the java client will automatically submit the cluster, so this step is optional.

1.2. Group consumption model

1.2.1. Group consumption model architecture diagram

Each group consumes all the data of the topic, and a message is sent to groupA and groupB.

1.2.2. Pseudo code of group consumption model:

The number of streams N: indicates that there are several consumer instances in a consumer group. In the above example, group A creates 2 streams and group B creates 4 streams.

1.2.3.consumer allocation algorithm

How to allocate kafka when the number of partitions is greater than the number of consumer instances in group A? here are the allocation steps:

1.3. Comparison of two consumption models

The Partition consumption model is more flexible but:

(1) you need to deal with all kinds of abnormal situations by yourself.

(2) you need to manage offset yourself (to achieve other semantics of message delivery)

The Group consumption model is simpler but inflexible:

(1) you do not need to handle exception cases or manage offset by yourself.

(2) you can only implement the default message delivery semantics of kafka at least once.

Knowledge supplement: three semantics of message delivery:

At least once (the message will not be lost, the messenger will get it at least once, but it may be repeated. After the producer sends it to the consumer, it will wait for the consumer to confirm, and if it does not receive the confirmation, it will be sent again) (the semantics implemented by kafka by default).

At most once, (the message will be lost)

Yes and only once.

1.4.java client parameter tuning

FetchSize: get a single packet size from the server

BufferSize: kafka client buffer size

Group.id: group name when grouping consumption (each specified group will get the full amount of data)

two。 Producer consumption model

Synchronous production model

Asynchronous production model

2.1. Synchronous production model

At least once successfully, send it to the kafka consumer

2.2. Asynchronous production model

Package and send it to kafka broker.

2.3. Pseudo-code description of two production models

Main ()

Create a connection to kafka broker: KafkaClient (host,port)

Select or customize the producer load balancing algorithm partitioner (algorithms are: hash, polling, random)

Set producer parameters (cache queue length, send time, synchronous / asynchronous parameter settings)

Construct Producer objects according to load balancing algorithm and set producer parameters

While True

GetMessage: get a message from upstream

Construct kafka messages according to the message format required by kafka

Get the partition according to the partition algorithm

Send a message

Handling exceptions

2.4. Comparison of two production models

Synchronous production model:

(1) low message loss rate

(2) High message repetition rate (reply confirmation has not been received due to network reasons)

(3) High latency (confirmation is required for each message sent)

(use in no-loss message scenarios)

Asynchronous production model:

(1) low latency

(2) High transmission performance; (500000 messages per second per partition)

(3) High message loss rate (without acknowledgement mechanism, if the sender queue is full, the message will be lost; the whole queue will be sent to)

(used in scenarios where messages are allowed to be dropped, one is lost occasionally)

2.5.java client code implementation (custom partition)

/ / synchronization configuration parameters:

Default serialization method: byte serialization.

Set partition algorithm: the default is the hash partition algorithm for key, and you can customize the partition algorithm.

Confirmation mechanism request.require.acks: reasonably set to 1; 0: never confirm that a copy of 1: leader receives this message and sends back confirmation-1: all copies of leader receive this message and send back confirmation

The message is sent in the form of key-value, and key must be set.

2.6.java client parameter tuning

Message.send.max.retries: number of failed retries to send

Retry.backoff.ms: the time when the delivery failed when no confirmation was received.

Producer.type: synchronous or asynchronous transmission

Batch.num.messages: cumulative maximum number of messages when sent asynchronously

Queue.buffering.max.ms: maximum cumulative time when asynchronously sent

Thank you for reading this article carefully. I hope the article "what are the programming models of kafka" shared by the editor will be helpful to you. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.