Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is Flume infrastructure". In daily operation, I believe many people have doubts about what Flume infrastructure is. Xiaobian consulted various materials and sorted out simple and easy operation methods. I hope to help you answer the doubts about "what is Flume infrastructure"! Next, please follow the small series to learn together!

1. Flume overview 1.1. Flume definition

Flume is a highly available, highly reliable, distributed system for mass log collection, aggregation, and delivery provided by Cloudera. Flume is based on streaming architecture, flexible and simple.

1.2 Flume infrastructure

1.2.1、Agent

An Agent is a JVM process that delivers data from source to destination in the form of events.

Agent consists of three parts: Source, Channel and Sink.

1.2.2、Source

Source is the component responsible for receiving data to Flume Agent. The Source component can handle log data of various types and formats, including avro, rift, exec, jms, spooling directory, netcat, sequence generator, syslog, http, legacy.

1.2.3、Channel

Channel is a buffer located between Source and Sink. Thus, Channel allows Source and Sink to operate at different rates. Channel is thread-safe and can handle several Source writes and several Sink reads simultaneously. Flume comes with two channels: Memory Channel and File Channel. (mainly Memory, general banking, insurance and other businesses need to use File to drop)

Memory channels are queues in memory. Memory Channel works in situations where data loss is not a concern. If data loss is a concern, Memory Channel should not be used because program death, machine downtime, or restart can lead to data loss.

File Channel writes all events to disk. So no data is lost in case of program shutdown or machine downtime.

1.2.4、Sink

The Sink constantly polls the Channel for events and removes them in batches, writing them in batches to a storage or indexing system, or sending them to another Flume Agent.

Sink component destinations include hdfs, logger, avro, thrift, ipc, file, HBase, solr, custom.

1.2.5、event

Flume The basic unit of data transmission that sends data from source to destination in the form of an Event. Event consists of two parts, Header and Body. Header is used to store some attributes of the event, which is a K-V structure. Body is used to store the data in the form of byte array.

At this point, the study of "what Flume infrastructure is" is over, hoping to solve everyone's doubts. Theory and practice can better match to help everyone learn, go and try it! If you want to continue learning more relevant knowledge, please continue to pay attention to the website, Xiaobian will continue to strive to bring more practical articles for everyone!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.