Shulou(Shulou.com)06/01 Report--

This article mainly shows you "Linux system how to install Flume", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "Linux system how to install Flume" this article.

Flume is a highly available, highly reliable, distributed mass log collection, aggregation and transmission system provided by Cloudera.

About flume: flume is a distributed log collection system produced by cloudera Software, which was donated to the apache Software Foundation in 2009 as one of the related components of hadoop.

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

It has a simple and flexible architecture based on streaming data streams. It has reliable reliability mechanism and many failure transfer and recovery mechanisms, and has strong fault tolerance and fault tolerance. It uses a simple extensible data model that allows applications to be analyzed online.

Detailed steps for installing flume on Linux system: premise: Hadoop has been set up

1 upload the compressed package to the virtual machine and decompress it

Tar-zxvf apache-flume-1.7.0-bin.tar.gz2 changes the name for ease of use

Mv apache-flume-1.7.0-bin flume3 modifies the flume-env.sh file:

Go to the conf directory under the flume installation directory

Cp flume-env.sh.template flume-env.sh vim flume-env.sh configures JAVA_HOME in it

4 configure the collection scheme according to the requirements of data collection, and describe it in the configuration file (the file name can be customized arbitrarily)

Here, create a new file under the conf directory of flume: vim netcat-logger.conf

Configure the following scenarios:

# define the names of the components in this agent a1.sources = R1 a1.sinks = K1 a1.channels = C1 # describe and configure source components: R1 a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # describe and configure sink components: K1 a1.sinks.k1.type = logger # describe and configure channel components Here a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100# describe and configure the connection between source channel sink a1.sources.r1.channels = C1 a1.sinks.k1.channel = c15 specify the collection scheme configuration file, and launch flume agent on the corresponding node to collect data

. / bin/flume-ng agent-c conf-f conf/netcat-logger.conf-n A1-Dflume.root.logger=INFO,console-c conf specifies the directory where the configuration file of flume itself is located

-f conf/netcat-logger.con specifies the acquisition scheme we describe

-n A1 specifies the name of our agent

6 Test

First, you need to send data to the port where agent collects and listens, so that agent can collect data.

On any machine that can connect to the agent node (I use the local machine here, and build a remote connection to the local machine)

Telnet anget-hostname port for example: telnet localhost 44444, but I do not have telnet installed in my virtual machine

II. Installation of telnet

1 check whether the rpm package of telnet-server is installed

If rpm-qa telnet-server has no output, it is not installed.

2 if not installed, install telnet-server

Yum install telnet-server3 detects whether the rpm package of telnet is installed

If rpm-qa telnet has no output, it is not installed.

4 if not installed, install telnet

Yum install telnet5 restarts the xinetd daemon

Service xinetd restart5 then executes again: telnet localhost 44444

Then you can see that flume has received the data to be collected.

How to quit telnet?

First press ctrl+] to exit to telnet >, then type telnet > quit to exit.

The above is all the contents of the article "how to install Flume in Linux system". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.