Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you what is the resource isolation of Kafka. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

What is the resource isolation of Kafka

Previously, due to the complexity and small scale of the business, big data platform for the division of Kafka clusters is relatively simple. As a result, after a period of time, the company's business data are mixed together, and the unreasonable use of a business topic may lead to some Broker overload, affecting other normal business, and even the risk that some Broker failures will affect the entire cluster and make the company-wide business unavailable.

In view of the above problems, two practices have been done in the transformation of the cluster:

Split independent clusters by functional attributes

Resource isolation with Topic granularity within a cluster

Cluster split

Split multiple Kafka physical clusters according to functional dimensions to isolate services and reduce the complexity of operation and maintenance.

In terms of the most important use of buried data, it is currently divided into three types of clusters. The functions of each cluster are defined as follows:

Log cluster: data collected from buried sites at each end will first land on the cluster, so this process cannot be interrupted due to Kafka problems, which requires high availability of Kafka. Therefore, the cluster will not provide subscriptions to ensure that consumers can control it. At the same time, the cluster business also serves as the source of offline collection, and the data will be dump to HDFS by hourly granularity through the Camus component, and this part of the data will participate in the subsequent offline calculation.

Full subscription cluster: most of the data in the cluster Topic is synchronized in real time from the Log cluster. As we mentioned above, the data of the Log cluster is not external, so the full cluster assumes the responsibility of consuming subscriptions. At present, it is mainly used in the real-time tasks within the platform to analyze the data of multiple business lines and provide analysis services.

Customized cluster: as mentioned earlier, we can split and merge data log sources according to the needs of the business side. At the same time, we also support customized Topic, which only needs to provide landing storage of Topic after shunting.

Resource isolation

The traffic size of Topic is an important basis for resource isolation within the cluster. For example, the two data sources with a large amount of logs in our business are the backend buried point data source server-event and the buried point mobile-event data source on the end. We want to avoid assigning topic partitions that store two data to nodes on the same Broker in the cluster. By physically isolating different Topic, you can avoid skew of traffic on the Broker.

This is what the resource isolation of Kafka shared by the editor is. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.