Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the application scenarios and usage of Flume". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn "what are the Flume application scenarios and usage?"

Introduction to Flume

Flume is a distributed and highly available mass log collection system. It supports the definition of the sender and receiver of log data, and can write log data from multiple sources to the specified receiver, such as text, HDFS, Hbase and so on.

I think what I admire most about Flume is that it can collect the log information of the system without intrusively without interfering with the operation of existing systems.

The data flow of Flume runs through event (events). The event is the basic data unit of Flume. It carries log data (in the form of a byte array) and header information, which are generated by Source outside the Agent. When a Source captures a pre-set event (in this case, an event in a broad sense), an event is generated and pushed to single or multiple Channel. You can think of Channel as a buffer that holds the event until Sink finishes processing it. Sink is responsible for processing event, which persists logs or transfers event to another Source.

Flume takes agent as the smallest independent running unit. Each machine runs an agent, and an agent is composed of Source, Sink and Channel components, as shown below:

Client production data runs in a separate thread, such as our application system.

Source collects data from Client and passes it to Channel.

Sink collects data from Channel and runs on a separate thread.

Channel connects sources and sinks, which is a bit like a queue.

Events can be logging, avro objects, and so on.

An agent contains multiple sources and sinks:

Application scenario and usage

Sources.type = netcat

Flume can listen to a computer (Client) to receive netcat and telnet messages from other computers, and then send these messages to designated places, such as hdfs, HBase, Kafka and so on.

Sources.type = exec

Flume can monitor the execution of an instruction on a computer (Client), and then send the output information generated by the execution of the instruction to a specified place, such as hdfs, HBase, Kafka. Such as setting:

Sources.command = tail-F / app/xxx.log

Then every time 10 new logs are generated, Flume will send the 10 new logs to the designated places, such as hdfs, HBase, Kafka and so on.

Sources.type = spooldir

Flume can listen for changes in a directory file on a computer (Client). When a new log file is generated, Flume sends the contents of the log file to a specified place, such as hdfs, HBase, Kafka, and so on.

Sources.type = http

Sources.type = syslogtcp

Flume can listen to the port of the TCP of a computer (Client) and send messages received from the port to a designated place, such as hdfs, HBase, Kafka, etc.

At this point, I believe that you have a deeper understanding of "Flume application scenarios and usage", might as well come to the actual operation of it! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.