Shulou(Shulou.com)05/31 Report--

This article will explain in detail what are the four stages of the evolution of Kafka. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Kafka is a popular message queue middleware, which can deal with massive data in real time, has the characteristics of high throughput, low delay and reliable message asynchronous transmission mechanism, and can well solve the problem of data exchange and transmission between different systems.

Four stages of the evolution of Kafka

The early big data platform introduced Kafka as a business log collection and processing system, mainly considering its high throughput and low latency, multiple subscriptions, data backtracking and other characteristics, which can better meet the needs of big data scenarios. However, with the rapid increase of business volume and the problems encountered in business use and system maintenance, such as imperfect registration mechanism and monitoring mechanism, the problems can not be located quickly, and the failure of some online real-time tasks leads to a backlog of messages, which challenges the stability and availability of Kafka clusters, and has experienced several serious failures.

It is urgent and difficult for us to solve the above problems. In view of some pain points in the use of Kafka on big data platform, we have done a series of practices from cluster use to application layer expansion, which includes four stages as a whole:

The first phase: version upgrade. Focusing on some bottlenecks and problems in the production and consumption of platform data, we selected the technology for the current version of Kafka and finally decided to use version 1.1.1.

The second stage: resource isolation. In order to support the rapid development of business, we have improved the construction of multi-clusters and the resource isolation between Topic in clusters.

The third stage: access control and monitoring alarm.

First of all, in terms of security, the early Kafka clusters were running naked. Because multiple product lines share Kafka, it is easy to cause data security problems due to misreading the Topic of other businesses. Therefore, we add the function of authentication based on SASL/SCRAM+ACL.

In terms of monitoring and alarm, Kafka has become the standard configuration of input data sources in real-time computing, so the backlog and throughput of Lag have become important indicators of the health of real-time tasks. Therefore, big data platform built a unified Kafka monitoring and alarm platform and named "radar" to monitor Kafka clusters and users in multiple dimensions.

The fourth stage: application expansion. In the process of opening up to the company's business lines in the early days, Kafka was not properly used by some business parties due to the lack of unified usage standards. In order to solve this pain point, we have built a real-time subscription platform, which enables the business side to apply for data production and consumption, user authorization, user monitoring and alarm, and many other links to automate the process through the form of application services. to create an overall closed loop from the use of the demand side to the omni-directional management and control of resources.

What are the four stages of Kafka evolution are shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.