Shulou(Shulou.com)05/31 Report--

This article will explain in detail what is the core practice of Kafka. The content of the article is of high quality, so the editor will share it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Core practice of Kafka

1. Version upgrade

Previously, big data platform has been using 0.8.3, the early version of Kafka, but up to now, the latest official Release version of Kafka has reached 2.3. therefore, many bottlenecks and problems gradually encountered in the process of using version 0.8 for a long time can be solved by version upgrade.

For example, here are some common problems when using older versions:

Lack of support for Security: data security issues and inability to use fine-grained management of resources through authentication and authorization

Brokerunderreplicated: found that broker is in underreplicated state, but is not sure the cause of the problem, it is difficult to solve.

The new feature cannot be used: such as transaction messages, idempotent messages, message timestamps, message queries, and so on.

The management of offset on the client depends on zookeeper, and the excessive use of zookeeper increases the complexity of operation and maintenance.

Monitoring metrics are not perfect, such as data size metrics of topic, partition and broker, and monitoring tools such as kafkamanager do not support the lower version of kafka well.

At the same time, some target version features are selected and investigated, such as:

Version 0.9, adding quotas and security, in which security authentication and authorization are the features that we are most concerned about

Version 0.10, a finer-grained timestamp. You can do a quick data search based on the offset to find the desired timestamp. It is very important to replay data based on Kafka data source in real-time data processing.

Version 0.11, idempotent and Transactions support and replica data loss / data inconsistency resolution.

Version 1.1, the improvement of operation and maintenance. For example, when ControllerShutDown wants to shut down a Broker, it takes a long and complicated process to get a lot of improvement in version 1.0.

In the end, version 1.1 was chosen because of the compatibility of Camus with the Kafka version and the comprehensive consideration of support for important new features in the use scenario. Here we briefly talk about the Camus component, which is also open source by Linkedin and is mainly used as an important way to Dump Kafka data to HDFS in our big data platform.

On what the core practice of Kafka is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.