Shulou(Shulou.com)05/31 Report--

This article shows you how to choose Kafka and Flume in the collection layer, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

The acquisition layer can mainly use Flume and Kafka technologies.

Flume:Flume is a pipeline flow mode, which provides many default implementations that allow users to deploy through parameters and extend API.

Kafka:Kafka is a persistent distributed message queue.

Kafka is a very general system. You can have many producers and many consumers share multiple theme Topics. By contrast, Flume is a dedicated tool designed to send data to HDFS,HBase. It has special optimization for HDFS and integrates the security features of Hadoop. Therefore, Cloudera recommends using kafka; if the data is consumed by multiple systems, using Flume if the data is designed for use by Hadoop.

As you know, Flume has many built-in source and sink components. However, Kafka obviously has a smaller ecosystem of production consumers, and Kafka's community support is not good. Hopefully this will improve in the future, but for now: using Kafka means you're ready to write your own producer and consumer code. If the existing Flume Sources and Sinks meet your needs, and you prefer a system that does not require any development, use Flume.

Flume can use interceptors to process data in real time. These are useful for data masking or overload. Kafka requires an external stream processing system to do this.

Both Kafka and Flume are reliable systems, and zero data loss can be guaranteed through proper configuration. However, Flume does not support copy events. So, if a node of the Flume agent crashes, even if you use a reliable file pipeline, you will lose these events until you recover the disks. If you need a highly reliable pipeline, then using Kafka is a better choice.

Flume and Kafka can be used together well. If your design requires streaming data from Kafka to Hadoop, it is also possible to use the Flume proxy and configure Kafka's Source to read the data: you don't have to implement your own consumers. You can directly take advantage of all the benefits of Flume's combination with HDFS and HBase. You can use Cloudera Manager to monitor consumers, and you can even add interceptors for some flow processing.

Flume and Kafka can be used together. Flume + Kafka is usually used. In fact, if you want to take advantage of Flume's existing ability to write HDFS, you can also use Kafka + Flume.

The above content is how to choose Kafka and Flume in the collection layer. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.