Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about the comparison of Flume, Kafka and NiFi in large data stream processing. Many people may not know much about it. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something from this article.

We will briefly introduce three Apache processing tools: Flume, Kafka, and Nifi. These three products have excellent performance, can scale horizontally, and provide plug-in mechanisms that can be expanded through custom components.

Apache Flume

A Flume deployment consists of one or more agents configured using a topology. The Flume agent is a JVM process that hosts the basic building blocks of the Flume topology, namely the source, channel, and receiver. The Flume client sends events to the source and places them in batches in a temporary buffer called channel from which data flows to the receiver connected to the final destination of the data. The sink can also be a subsequent data source for other Flume agents. Agents can be linked, and each agent has multiple sources, channels, and receivers.

Flume is a distributed system that can be used to collect, aggregate, and transfer stream events to Hadoop. It has many built-in sources, channels, and receivers, such as Kafka channels and Avro receivers. Flume is configuration-based and has interceptors to perform simple transformations on the data in the channel.

It is easy to lose data when using Flume if you are not careful. For example, the disadvantage of choosing a memory channel for high throughput is that data is lost when the proxy node is shut down. The file channel will provide persistence at the expense of increased latency. Even so, because the data is not replicated to other nodes, the file channel is only as reliable as the underlying disk. Flume provides scalability through multi-hop / fan-in fan-out streams. For high availability (HA), agents can scale horizontally.

Apache Kafka

Kafka is a distributed high-throughput message bus that separates data producers from consumers. Messages are organized by topic, which is divided into multiple partitions that replicate between nodes in the cluster (called agents). Compared with Flume, Kafka has better scalability and message persistence. Kafka now has two styles: one is the "classic" producer / consumer model, and the other is the new Kafka-Connect, which provides configurable connectors (source / receiver) for external data stores.

Kafka can be used for event handling and integration between large software system components. In addition, kafka comes with kafka streams, which can be used for simple flow processing without the need for separate clusters, such as apache spark or apacheFlink.

Because messages are persisted on disk and replicated in the cluster, data loss is not as common as Flume. That is, producers / sources and consumers / receivers usually need custom coding, whether using a Kafka client or through Connect API. Like Flume, there is a limit to the size of messages. Finally, in order to be able to communicate, the producers and consumers of Kafka must agree on the agreement, format, and architecture, which may be problematic in some cases.

Apache NiFi

Unlike Flume and Kafka, NIFI can handle messages of any size. Behind the Web-based drag-and-drop user interface, NIFI runs in the cluster and provides real-time control so that you can easily manage data movement between any source and any destination. It supports distributed and distributed sources of different formats, patterns, protocols, speeds, and sizes.

NiFi can be used in mission-critical data streams with stringent security and compliance requirements, where we can visualize the entire process and make changes in real time. As of this writing, it has nearly 200 ready-to-use processors (including Flume and Kafka processors) that can be dragged and dropped, configured, and put into use immediately. Some of the key features of NiFi are priority queuing, data tracking, and back pressure threshold configuration for each connection.

Although NiFi is used to create fault-tolerant production pipes, it does not replicate data like Kafka. If a node fails, the flow can be directed to another node, but the data queued for the failed node must wait for the node to recover. NiFi is not a mature ETL tool and is not suitable for complex computing and event processing (CEP). To do this, it should connect to a streaming framework, such as Apache Flink,Spark Streaming or Storm.

Combination

No tools meet all your requirements. Combining tools that perform different operations in a better way can enhance functionality and increase the flexibility to handle more scenarios. Depending on your needs, NiFi and Flume can act as Kafka producers or consumers.

Flume-Kafka integration is so popular that it has its own name: Flafka (I didn't do that). Flafka includes Kafka source, Kafka channel and Kafka pool. The combination of Flume and Kafka,Kafka can avoid custom coding and take advantage of Flume's hands-on resources and receivers. Flume events through the Kafka channel will be stored and replicated in the Kafka agent for resilience.

The combination tool may seem wasteful because it seems to overlap in functionality. For example, both NiFi and Kafka provide agents to connect producers and consumers. However, they behave differently: in NiFi, most of the data flow logic is not in the producer / consumer, but in the agent, allowing centralized control. NiFi is built to do one important thing: data flow management. Through the combination of the two tools, NiFi can take full advantage of Kafka's reliable streaming data storage while solving the data flow challenges that Kafka cannot solve.

After reading the above, do you have any further understanding of the comparison of Flume, Kafka and NiFi in large data stream processing? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.