Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what is the use of log collection tool Flume", the content is simple and easy to understand, the organization is clear, I hope to help you solve doubts, let Xiaobian lead you to study and learn "what is the use of log collection tool Flume" this article.

Flume is a distributed, reliable, and highly available system for mass log collection, aggregation, and transmission. Support customization of various data senders in the log system for collecting data; at the same time, Flume provides the ability to simply process data and write to various data receivers (such as text, HDFS, Hbase, etc.). Flume's data flow is traversed by events. Events are the basic data units of Flume, which carry log data (byte array form) and carry header information. These events are generated by Source outside Agent. When Source captures events, it will be formatted specifically, and then Source will push events into Channel (single or multiple). You can think of a Channel as a buffer that will hold an event until Sink finishes processing it. Sink is responsible for persisting logs or pushing events to another Source.

Some of Flume's core concepts:

1. Agent: Use JVM to run Flume. Each machine runs an agent, but you can have multiple sources and sinks in an agent.

2. Client: Produces data, running in a separate thread.

3. Source: Collect data from clients and pass it to Channel.

4. Sink: Collect data from Channel and run in a separate thread.

5. Channel: Connect sources and sinks, which is a bit like a queue.

6. Events: can be log records, avro objects, etc.

Flume takes agent as the smallest independent operating unit. An agent is a JVM. A single agent consists of three components: Source, Sink and Channel, as shown below:

Notably, Flume provides a large number of built-in Source, Channel, and Sink types. Different types of Source,Channel and Sink can be combined freely. The combination method is based on the profile set by the user, which is very flexible. For example, Channel can store events temporarily in memory or persist them to local hard disk. Sink can write logs to HDFS, HBase, or even another Source, etc. Flume supports users to establish multi-level flows, that is, multiple agents can work together, and supports Fan-in, Fan-out, Contextual Routing, Backup Routes, which is exactly where NB is. As shown below:

Log collection practical application cases:

Flume: Log Collection

HDFS/HBase: Log Storage

Hive: Log analysis

The above is "What is the use of log collection tool Flume" All the contents of this article, thank you for reading! I believe that everyone has a certain understanding, hope to share the content to help everyone, if you still want to learn more knowledge, welcome to pay attention to the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.