Shulou(Shulou.com)06/01 Report--

This article shares with you what are the important configuration parameters of the Kafka production environment. Xiaobian thinks it is quite practical, so share it with you to learn. I hope you can gain something after reading this article. Let's not say much. Let's take a look at it together with Xiaobian.

Kafka has great advantages in terms of resilience, fault tolerance and high throughput. To achieve the optimal production environment, to play these characteristics, we need to carry out a series of configurations. Kafka provides a lot of configuration properties that can be confusing for beginners. In fact, most of the configuration has met most of the use scenarios, Xiaobian shared a summary of several more important configuration parameters, mainly for the configuration of the producer side.

acks

The acks parameter specifies how many partition copies must receive the message before the producer considers the message to be written successfully. This parameter plays an important role in whether the message is lost. The configuration of this parameter is as follows:

acks=0, indicating that the producer does not wait for any response from the server before successfully writing the message. In other words, once a problem occurs and the server does not receive the message, the producer has no way of knowing and the message is lost. Since there is no need to wait for a response from the server, the reconfiguration can send messages at the maximum speed supported by the network, thus achieving high throughput.

acks=1, which means that as long as the leader partition copy of the cluster receives the message, it will send a successful response ack to the producer. At this time, the producer can consider the message to be successfully written after receiving the ack. Once the message cannot be written to the leader partition copy (e.g. network reasons, leader node crash), the producer receives an error response, and when the producer receives the error response, it resends the data to avoid data loss. The throughput of this mode depends on whether asynchronous or synchronous transmission is used.

Scream alert: if the producer receives an error response, even if it resends the message, it is still possible to lose data. For example, if a node that did not receive a message becomes the new Leader, the message will be lost.

acks =all, indicating that the producer will receive a response from the server only if all participating nodes (copies of the ISR list) receive the message. This pattern is the highest level and safest, ensuring that more than one Broker receives the message. Latency in this mode can be high.

min.insync.replicas

As mentioned above, when acks=all, all copies need to be synchronized to send a successful response to the producer. There is a question here: What happens if the Leader copy is the only synchronous copy? This is equivalent to acks=1. So it's unsafe.

Kafka's Broker provides a parameter min.insync.replicas, which controls how many copies the message is written to before it is considered "true writing." The default value is 1. Setting a value greater than 1 in the production environment can improve the persistence of the message. Because if the number of synchronized copies falls below this configured value, the producer receives an error response, ensuring that messages are not lost.

replica.lag.time.max.ms

In-sync replica(ISR) is called synchronous replica, copies in ISR are synchronized with the Leader, so followers not in the list are considered to be out of sync with the Leader. So, what kind of copy exists in ISR? The first thing to be clear is that a copy of Leader always exists in ISR. Whether the follower copy is in ISR depends on whether the follower copy is "synchronized" with the Leader copy.

Scream tip: The understanding of "whether the follower copy is synchronized with the Leader copy" is as follows:

(1)The synchronization mentioned above does not mean complete synchronization, that is, it does not mean that once the follower copy lags behind the Leader copy, it will be kicked out of the ISR list.

(2)Kafka's broker has a parameter ** replica.tag.time.max.ms **, which indicates the longest time interval between the follower replica and the Leader replica, and the default is 10 seconds. This means that as long as the time interval between the follower copy and the leader copy does not exceed 10 seconds, the follower copy and the leader copy can be considered synchronized, so even if the current follower copy lags behind the Leader copy by a few messages, as long as it catches up with the Leader copy within 10 seconds, it will not be kicked out.

(3) If the follower copy is kicked off the ISR list, it will be added to the ISR list again when it catches up with the Leader copy, so the ISR is a dynamic list, not a static one.

retries

The error that the producer receives from the server may be temporary (e.g. partition cannot find leader). In this case, the value of the retries parameter determines the number of times the producer can resend the message, and if this number is reached, the producer will give up trying again and return an error. By default, producers wait 100ms between each retry, and the interval can be configured via the retry.backoff.ms parameter.

For example, acks=all and min.insync.replicas=2 are set. For some reason, all followers are down, and because min.insync.replicas=2, producers cannot receive ack from Broker.

At this point we receive an error message from the Producer side: "Broker: Not enough in-sync replicas". This means that Kafka cannot append messages (data) to the Broker because there are not enough ISRs. At this point, the following error message will appear on the Broker side:

org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(0) is insufficient to satisfy the min.isr requirement of 2 for partition

By default, Producer doesn't handle this error, which results in message loss, i.e. **at-most-once ** semantics. We can configure the retry count to have producers resend messages. For example, configure retries=3, default is 0

enable.idempotence

In some cases, the message is actually submitted to all synchronous replicas, but the Broker cannot send an acknowledgement ack to the Producer due to network problems. Since we set retries=3, the producer will resend the message 3 times, which may cause duplicate messages in the topic.

For example, a producer sends 1M messages to the topic, and the broker fails after submitting the message but before the producer receives all the acknowledgements. In this case, due to the retry mechanism, more than 1M messages may eventually be received on the topic, which is also known as at-lease-once semantics.

Of course, we want to implement exactly-once semantics, which means that even if the producer resends the message, the consumer should only receive the same message once.

At this point, idempotent operations are required, which means that performing an operation once or performing an operation multiple times has the same effect. Configuration idempotent is very simple, by configuring enable. idempotency =true, the default is false.

So how is idempotent achieved? Since messages are sent in batches, each batch has a sequence number. On the Broker side, the maximum sequence number of each partition is tracked. If a batch with a lower or equal sequence number occurs, broker will not write the batch to topic. In this way, in addition to ensuring idempotencies, batch order can also be ensured.

max.in.flight.requests.per.connection

This parameter specifies how many messages the producer can send before receiving a response from the server. The higher its value, the more memory it consumes, but it also increases throughput. Setting it to 1 ensures that messages are written to the server in the order they were sent, even if retries occur.

Because if two batches are sent to a single partition, and the first batch fails and is retried, but then the second batch is written successfully, records in the second batch may appear first, and thus out-of-order occurs.

If idempotent is not enabled, but you still want to send messages sequentially, you should configure this setting to 1. However, if idempotent is already enabled, there is no need to explicitly define this configuration.

buffer.memory

This parameter is used to set the size of the producer memory buffer that producers use to buffer messages to be sent to the server. If an application sends messages faster than it can to the server, it will run out of producer space. At this point, the send() method call either blocks or throws an exception, depending on how max.block.ms is set.

When send() is called by the producer, the message is not sent immediately, but is added to the internal buffer. The default buffer.memory value is 32MB. If the producer is sending messages faster than it can send messages to the broker, or if there is a network problem, the send() method call is blocked as often as configured by the max.block.ms parameter, which defaults to 1 minute.

max.block.ms

This parameter specifies the blocking time of the producer when the send() method is called or the partitionsFor() method is used to get metadata. These methods are blocked when the producer's send buffer is full or no metadata is available. When the blocking time reaches max.block.ms, the producer throws a timeout exception.

linger.ms

This parameter specifies how long the producer waits for more messages to be added to the batch before sending it. The kafka producer sends out the batch when the batch is full or linger.ms reaches its upper limit. By default, producers send messages whenever threads are available, even if there is only one message in the batch. Set linger.ms to a number greater than 0 to make producers wait a while before sending batches, allowing more messages to be added to the batch. While this increases latency, it also increases throughput (because more messages are sent at once, the overhead per message is smaller).

batch.size

When multiple messages need to be sent to the same partition, the producer puts them in the same batch. This parameter specifies the amount of memory a batch can use, measured in bytes (not messages). When the batch is full, all messages in the batch are sent out. However, producer wells may not wait until the batch is full, depending on the configuration of linger. ms. For example, if the linger.ms time is up, even if the batch contains only one message, it will be sent immediately. So even if the batch size is set very large, it will not cause delay, just use more memory. But if it's set too small, it adds some extra overhead because producers need to send messages more frequently.

You can use linger.ms and batch.size configurations. linger.ms is the delay time before the batch is ready to be sent, the default value is 0. This means that even if there is only 1 message in the batch, the batch will be sent immediately. Sometimes, linger.ms is increased to reduce the number of requests and improve throughput. But this will cause more messages to remain in memory. batch.size is the maximum size of a single batch, and the batch will be sent when either of these two requirements is met.

compression.type

By default, messages are not compressed when sent. This parameter can be set to snappy, gzip, or lz4 and specifies which compression algorithm is used to compress the message before it is sent to the broker. Using compression reduces network transmission overhead and storage overhead, which are often bottlenecks in sending messages to Kafka.

The above is what the important configuration parameters of Kafka production environment are. Xiaobian believes that some knowledge points may be seen or used in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.