Shulou(Shulou.com)06/01 Report--

How to determine the appropriate number of partitions for the Kafka cluster and the disadvantages caused by too many partitions? in view of this problem, this article introduces the corresponding analysis and solutions in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

One of the reasons for the high throughput of Kafka is to save messages in topic to different broker in the Kafka cluster through partition. Both producer of Kafka and consumer can manipulate partition in topic concurrently, so partition is the minimum unit of Kafka parallelism tuning.

In theory, the more partitions a topic has, the more throughput the entire cluster can theoretically achieve.

But is it true that the more partitions Kafka topic is configured in actual production, the better? Obviously not!

Too many partitions have the following disadvantages:

First, the more memory the client / server needs to use

After Kafka0.8.2, there is a parameter batch.size in the client-side producer, which defaults to 16KB. It caches messages for each partition, and when the data accumulates to a certain size or enough time, the accumulated messages will be removed from the cache and sent to the broker node. This feature is designed to improve performance, but as the number of partitions increases, so does this part of the cache.

At the same time, the memory footprint of the server when consuming messages and the number of processor threads opened to achieve higher throughput performance will also increase with the increase in the number of partitions. For example, if there are 10000 partitions and the number of Socket threads needs to match the number of partitions (in most cases, it is the best consumer throughput configuration), then 10000 threads will be created in consumer client, 10000 threads will be created in consumer client, and about 10000 threads will need to be created to obtain partition data. The overhead cost of threads is obviously not to be underestimated!

In addition, the overhead on the server side is not small. If you read the Kafka source code, you can find that many components on the server side maintain partition-level caches in memory, such as controller,FetcherManager, etc., so the more the number of partitions, the greater the cost of this cache.

Second, the cost of file handles

In Kafka's broker, each partition corresponds to a directory of the disk file system. In the Kafka data log file directory, each log data segment is assigned two files, an index file and a data file. In the current version of kafka, each broker opens an index file handle and a data file handle for each log segment file. Therefore, as the number of partition increases, the more file handles you need to keep open, which may eventually exceed the number of file handles configured by the underlying operating system.

Third, more partitions may increase end-to-end latency

The Kafka end-to-end delay is defined as the time it takes for the producer side to publish the message to the consumer side to receive the message, that is, the time that the consumer receives the message minus the time that the producer publishes the message.

Kafka exposes the message to the consumer only after the message is submitted. For example, the message is not exposed until all in-sync replica lists are replicated synchronously. Therefore, the time it takes for in-sync replica replication will be the most important part of the end-to-end latency of kafka. By default, when each broker replicates data from another broker node, the broker node assigns only one thread to this work, which needs to complete the replication of all partition data from that broker.

Note that the above problems can be mitigated by increasing the kafka cluster. For example, there is a difference in latency between 1000 partition leader on one broker node and 10 broker nodes. In a cluster of 10 broker nodes, each broker node needs to handle an average of 100 partitions of data replication. At this point, the end-to-end delay will change from tens of milliseconds to just a few milliseconds.

As a rule of thumb, if you are concerned about message latency, it is a good idea to limit the number of partition per broker node: for b broker nodes and kafka clusters with a replication factor of r, the number of partition in the entire kafka cluster should not exceed 100*b*r, that is, the number of leader in a single partition should not exceed 100.

Fourth, reduce high availability

Kafka achieves high availability and stability of Kafka clusters through multi-replica replication technology. Each partition has multiple copies of the data, each of which exists in a different broker. Of all the data replicas, one is leader and the other is follower.

Within the Kafka cluster, all data replicas are managed in an automated way and ensure that the data of all data replicas are synchronized. Requests from both the producer side and the consumer side to the partition are processed through the broker where the copy of the leader data is located. When the broker fails, all partition of the leader data copy in that broker will become temporarily unavailable. Kafka will automatically select a leader from other copies of the data to receive requests from the client. This process is done automatically by the Kafka controller node broker, mainly to read and modify some metadata information of the affected partition from the Zookeeper.

Normally, when a broker stops the service in a planned way, the controller removes all the leader on the broker one by one before the service stops. Since the movement time of a single leader takes only a few milliseconds, from a customer perspective, planned service downtime will only make the system unavailable in a very small time window. (note: during a planned downtime, only one leader will be transferred in each time window of the system, and all other leader will be available. )

However, when broker stops service unplanned (for example, in kill-9 mode), the system's unavailable time window will be related to the number of partition affected. Suppose there are 2000 partition in a 2-node kafka cluster, and each partition has 2 copies of data. When one of the broker went down unplanned, all 1000 partition became unavailable at the same time. Assuming that each partition recovery time is 5ms, then the recovery time of 1000 partition will take 5 seconds. Therefore, in this case, the user will observe that the system has an unavailable time window of 5 seconds.

If the broker where the outage occurs happens to be the controller node: in this case, the election process for the new leader node will not start until the controller node returns to the new broker. Error recovery for controller nodes will occur automatically, but the new controller node needs to read the metadata information for each partition from the zookeeper to initialize the data. For example, assuming that there are 10000 partition in a Kafka cluster, and each partition costs about 2ms to recover metadata from zookeeper, the recovery of controller will increase the unavailability window by about 20 seconds.

To sum up, in general, the more partition in the Kafka cluster, the higher the throughput. However, if the total amount of partition in the Kafka cluster is too large or there are too many partition in a single broker node, it may have a potential negative impact on system availability and message delay, which needs our attention.

So how to determine a reasonable number of partitions?

Achieving load balancing at the partition level is the key to achieving throughput. The appropriate number of partition can achieve the purpose of highly parallel read and write and load balancing, which needs to be estimated according to the target throughput of producers and consumers of each partition.

You can follow certain steps to determine the number of partitions: determine the initial value of the partition based on experience such as the amount of data that a topic "receives" every day, and then test the producer throughput and consumer throughput of the topic. Suppose their values are Tp and Tc, respectively, and the unit can be MB/s. Then assume that the total target throughput is Tt, then numPartitions = Tt / max (Tp, Tc)

Description: Tp represents the throughput of producer. Testing producer is usually easy because its logic is very simple, just send a message directly to Kafka. Tc represents the throughput of the consumer. Testing Tc usually has more to do with what is processed after the application consumes the message, which is relatively complicated. This is the answer to the question about how to determine the appropriate number of partitions for the Kafka cluster and the disadvantages caused by too many partitions. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.