Shulou(Shulou.com)06/01 Report--

In view of the pain points, advantages and applicable scenarios of message queuing service Kafka, this article introduces the corresponding analysis and solutions in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Abstract: message queuing Kafka is a distributed, high-throughput and highly scalable message queuing service, which is widely used in log collection, monitoring data aggregation, streaming data processing, online and offline analysis, etc., and is one of the indispensable products in big data's ecology.

Message queuing Kafka

Message queuing Kafka is a distributed, high-throughput and highly scalable message queuing service. Compared to Apache Kafka, message queuing Kafka provides a fully managed service. It is also briefly introduced here that Apache Kafka,Apache Kafka is a distributed push-subscribe-based messaging system, which is fast, scalable and persistent. It is now an open source system under Apache and is widely used in big data scenarios as part of the hadoop ecosystem.

Message queuing Kafka provides fully managed services for Apache Kafka, which completely solves the pain point of open source products for a long time. Users only need to focus on business development, do not need to deploy operations and maintenance, low-cost, more flexible, more reliable. The most important feature of message queuing product is fully managed service, which mainly involves two features: compatibility and convenience. First of all, for compatibility, message queuing Kafka is 100% compatible with Apache Kafka, and for users, open source clients in various languages can access seamlessly. Users who currently use open source Kafka can use message queuing Kafka products only by changing one access point. At the same time, message queuing Kafka is compatible with all the ecologies of Apache Kafka. Secondly, for convenience, message queuing Kafka does not need to be deployed. Users only need to enter instance information after purchasing message queuing Kafka, and they can use the service of message queuing Kafka within 15 minutes, so it is very convenient and easy to use.

The above is the overall introduction of message queuing Kafka, which will be divided into three modules: pain points, advantages and scenarios to share with you in more detail. First of all, we will share with you the user pain points collected by Aliyun for the message queuing service, and what are the advantages of message queuing Kafka according to these pain points. Finally, we will introduce the scenarios applicable to message queuing Kafka.

Pain Point: the annoyance of self-built Kafka

It is difficult to operate and maintain Apache Kafka.

For Kafka, from the user's point of view, it is a very simple product that provides a publish and subscribe model. Then, in terms of operation and maintenance of Kafka, it will be very difficult, because it needs to pay attention not only to similar roles like broker and controller in the entire cluster, but also to some products it depends on, such as ZooKeeper. Therefore, the operation and maintenance of the above modules not only involves parameter tuning, but also faces problems such as capacity expansion and reduction with the growth of business. In addition, you need to pay attention to the disk and network conditions. Therefore, to sum up, the cost and difficulty of operation and maintenance of self-built Kafka are very great. Next, I would like to share with you some specific examples.

Data confusion

Some users reported that they had data confusion when using Kafka clusters. As we all know, there are Controller and Broker roles in the Kafka cluster. If an exception occurs in the Controller, a Broker is automatically selected from the Broker to become the new Controller. However, due to abnormal conditions such as the network, the Controller that died at the beginning may be resurrected, and after the resurrection, there will be a "brain fissure" for the whole cluster. Because the main responsibility of Controller is to manage the partition and replica status of the entire cluster, and when there is a "brain crack", it will cause data confusion, which is unacceptable to users.

ZooKeeper is not available

The whole Kafka cluster is strongly dependent on ZK, and the operation and maintenance work of ZooKeeper is also huge and complex. For example, when operation and maintenance personnel do not know much about ZooKeeper, they may not know how to deploy ZooKeeper or how to ensure that ZK will be available in the same or multiple data center, which often requires consideration and tradeoff by operation and maintenance personnel. Important data of Kafka will be stored on ZooKeeper. When ZK is not available, the disaster recovery group and stored data of the entire cluster will be affected.

Bandwidth concern

For users, when building their own Kafka, they not only need to pay attention to its peripheral dependent products, but also need to pay attention to a problem often encountered within the cluster-bandwidth. From the user's point of view, it is often necessary to make a trade-off against the number of copies. In order to improve the reliability and disaster recovery capability, the cluster often needs three replicas, and when the number of replicas is large, it will involve data replication between machines, which will increase the bandwidth of the network. At the same time, because the Broker is peer-to-peer, and the data needs to be synchronized from the Controller. In this way, Controller not only needs to undertake its own tasks, but also needs to provide services, and as far as its own design is concerned, these two parts of tasks are not prioritized, so in the case of large cluster scale, it will cause network bandwidth congestion. Aliyun message queue Kafka has already helped users solve the above problems. Users do not need to make a trade-off between backups. Aliyun will help users achieve three copies of data storage and make the service availability up to 99.9%.

Disk operation and maintenance

User-built Kafka will also encounter some other problems, such as disk operation and maintenance problems. Since version 0.110, Consumer offsets is not only stored on the ZK side, it can be stored as a normal Topic in the Kafka cluster. The retention policy of the entire Consumer offsets determines the disk footprint, so it is possible that the disk footprint is too high because the wrong parameters are set. At the same time, what users often see is that their cluster has 100T disks, and it has become unwritable after using only a few tens of T. As we all know, there are two ways to partition data in Producer. Hash may tilt the Hash, and the use of RoundBobin may also lead to uneven disk footprint. For users, what they may see is that the user has bought a lot of disks and the disks are not full, but Producer is no longer writable. With regard to the details of disk operation and maintenance, message queuing Kafka has already solved the problem for users.

Data loss

In fact, for users, the most distressing is the problem of data loss. Kafka provides users with three data storage strategies, the first can be regarded as OneWay mode, the second is equivalent to putting a backup data on disk, and the last one is equivalent to putting all backup data on disk in order to succeed. The selection process of these three ways is actually a game between usability and performance. When the network load is high or the disk is difficult to write, it may cause disk write failure. At the same time, Kafka data is initially stored on PageCache and periodically brushed to disk, but not every message sent successfully is stored on disk. If there is a power outage or machine failure, the data stored in memory will be lost. In addition, there is also a situation where data will be lost when the amount of data in a single batch exceeds the limit. Using message queue Kafka, users do not need to do the game and consideration of the selection of these data, because as long as the message queue Kafka sends data successfully, then the data will be persisted to ensure that the data will not be lost. Because message queuing Kafka makes these optimizations, the reliability of the data can reach 8 9s (that is, 99.999999%).

Advantages of message queuing Kafka

What I share with you above is the advantages of message queuing Kafka. Let's sum it up again. Message queue Kafka is fully compatible with Apache Kafka, and the whole ecological products that Apache Kafka can use, such as Flume at the upper end and Spark, Storm, Flink and ES at the lower end, are also fully compatible with message queue Kafka. Secondly, message queuing Kafka provides fully managed services, that is, no matter whether there are disk problems or network problems in the cluster, any problems caused by Kafka itself or the products it depends on are solved by a professional team. For users, what they can see is 99.9% availability of the product and can bring a very stable state to users, while the underlying technical details are handled by Aliyun's professional team. For the parts of high availability and high reliability, it is strongly related to full hosting. For the reliability of data, it is the most important for every product, because when the loss of data occurs, it may make the whole business logic wrong, and then cause some major failures. What Aliyun promises is that when users use message queue Kafka to send messages, as long as the returned messages are successful, then the reliability of the data can reach 8 9s, which users do not need to worry about. At the same time, the Aliyun message team provides users with very practical business reports and a flexible and comprehensive business monitoring system, and the business monitoring and reporting is based on the user's business dimension. All the business-related indicators concerned by users, including the disk water level, Topic and Consumer Group of the entire cluster, will be deposited in the console of the message queue Kafka. Users can see the operation of the overall business by logging in to the console directly. Finally, the data running on the message queue Kafka is very secure. Through the isolation, authentication, encryption and blacklist of the VPC network, users' data can be guaranteed to be very secure. At the same time, one of the great advantages of message queuing Kafka is that every instance it buys is purchased exclusively by the user. The interaction between users will not lead to the instability of the whole system.

Scene

The above introduces the advantages of message queuing Kafka, and then shares its applicable scenarios. In fact, it can be considered that message queuing Kafka and open source Apache Kafka are suitable for the same scenario, except that message queuing Kafka has higher reliability and availability, and does not require users to carry out their own operation and maintenance.

Build a log analysis platform

Taobao, Tmall platform and other companies produce a large number of logs every day. Operations, operation and maintenance teams and some decision makers need to analyze and count the entire log data. The performance of Kafka itself is very efficient, and the characteristics of Kafka determine that it is very suitable to be a "log collection center". This is because Kafka is not aware of the business when collecting logs, it is compatible with its own upstream, and can directly configure encrypted messages. When the log data is sent to the Kafka cluster, it is completely non-invasive to the business. At the same time, it can directly connect offline warehouse storage such as Hadoop/ODPS and Strom/Spark to realize real-time online analysis downstream. In this case, using Kafka, users only need to pay attention to the business logic in the whole process, and can realize statistics, analysis and reporting without doing more development.

Site activity tracking scenario

In addition to implementing data analysis to form reports, Kafka can also implement website activity tracking scenarios. Through Kafka, we can collect the activity data of the website in real time, such as users' browsing, searching and behavior of the page. Message queuing Kafka can be used to segment different data models above the business through Topic. In that case, users can split according to registration, login, purchase, and so on. For different scenarios to be tracked downstream, you can interface with different processing systems, such as real-time processing, real-time monitoring and offline processing. Kafka is very convenient and easy to use in this scenario.

Data generates value in flow

The first two examples are that message queue Kafka plays the role of data input stream in the whole solution, while Kafka can not only act as data input stream, but also do stream calculation, such as stock market trend analysis, meteorological data measurement and control, website user behavior analysis and other fields, because of the fast data generation, strong real-time performance and large amount of data in these fields. Therefore, it is difficult to unify the collection and storage before processing, which leads to the traditional data processing architecture can not meet the needs. With the emergence of stream computing engines such as Kafka Stream and Storm/Samza/Spark, the data can be calculated and analyzed according to business requirements, and finally the results can be saved or distributed to the required components.

Multiplex forwarding

The scenario that we often encounter is that different calculation methods are required for different business dimensions, for example, real-time flow processing may be required for reconciliation systems, and batch computing may be used for statistical analysis. The use of Kafka can achieve multi-forwarding, upstream production of a piece of data, multiple downstream nodes can obtain this data and make corresponding processing, so Kafka can complete the function of data multi-forwarding.

This is the answer to the question about the pain points, advantages and applicable scenarios of the message queuing service Kafka. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.