Shulou(Shulou.com)06/01 Report--

Cruise Control enhances the example analysis of Kafka load balancing. I believe many inexperienced people don't know what to do about it. Therefore, this article summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

Cruise Control is a Kafka load balancing component that can be installed and used in large Kafka clusters. When adding or removing Kafka agents, Cruise Control can automatically balance partitions based on specific conditions.

The architecture of Cruise Control consists of a load monitor, an analyzer, an exception detector, and an executor, as shown in the following figure.

The load monitor generates a cluster workload model based on standard Kafka metrics and resource metrics to take advantage of disk, CPU, byte input rate and byte output rate. Enter the cluster model into the exception detector and analyzer.

The analyzer generates optimization recommendations according to the optimization objectives provided by the user and the cluster workload model from Load Monitor. Hard goals and soft goals can be set. Hard goals must be achieved, but if hard goals are achieved, soft goals will not be achieved. If the optimization result violates the hard goal, the optimization will fail.

The exception detector is responsible for detecting the following anomalies:

The reason for the abnormal phenomenon results in the failure of Broker. Non-empty Broker crashes or leaves the cluster. Cruise Control repairs the cluster by removing the faulty broker. Violation of target optimization is violated. If self-healing is enabled, Cruise Control automatically analyzes the workload and executes tuning recommendations. Disk failure is not an empty disk death. If self-healing is enabled, Cruise Control moves all offline copies to the normal agent. The index collected by the indicator exception is abnormal. It may be degraded first, and if the exception still exists, delete the slow broker according to the self.healing.slow.brokers.removal.enabled configuration.

The executor executes the optimization recommendation, which can be safely interrupted when it is executed. Execution is always a resource-aware process.

Retrieval metrics Cruise Control uses Kafka to store and retrieve its metrics, which are read by Metric Fetcher.

In the load monitor, Metric Fetcher Manager is responsible for coordinating all sampling tasks: Metric sampling task, Bootstrap task and linear model training task. Each sampling task is performed by a configured number of Metric Fetcher threads. Each Metric Fetcher thread uses a pluggable Metric Sampler to get the sample. Each Metric Fetcher allocates several partitions in the cluster to get samples. The indicator standard sample is organized by the indicator standard sample aggregator, which puts each indicator standard sample into the workload snapshot according to the time stamp of the indicator standard sample. The cluster workload model is the main output of the load monitor. The cluster workload model reflects the current replica allocation of the cluster and provides an interface for moving partitions or replicas. The analyzer uses these interfaces to generate optimization solutions. The sample storage area stores indicator standards and training samples for future use. With metrics standard samplers, you can deploy Cruise Control to a variety of environments and use it with existing metrics standards systems. When using Cruise Control in a Cloudera environment, HttpMetricsReporter metrics are reported to the Cloudera Manager time series database. As a result, Kafka metrics can be read using Cloudera Manager. After reading the above, have you mastered the method of Cruise Control to enhance the example analysis of Kafka load balancing? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.