Shulou(Shulou.com)06/02 Report--

This article mainly introduces how to install and deploy Flume, which has a certain reference value, and interested friends can refer to it. I hope you will gain a lot after reading this article.

Environment:

1.flume-ng-1.6.0-cdh6.7.0.tar.gz

2.jdk 1.8

1. Download the installation package and unzip it

Wget http://archive.cloudera.com/cdh6/cdh/5/flume-ng-1.6.0-cdh6.7.0.tar.gz

Tar-xvf flume-ng-1.6.0-cdh6.7.0.tar.gz

Rm flume-ng-1.6.0-cdh6.7.0.tar.gz

Mv apache-flume-1.6.0-cdh6.7.0-bin flume

two。 Configure environment variables

Vi / etc/profile

Export FLUME_HOME=/opt/software/flume

ExportPATH=.:$FLUME_HOME/bin:$KAFKA_HOME/bin::$ZOOKEEPER_HOME/bin:$PATH

Source / etc/profile

3. Configure the flume-env.sh file

Cd / opt/software/flume/conf/

Cp flume-env.sh.template flume-env.sh

Vi flume-env.sh

Export HADOOP_HOME=/opt/software/hadoop

Export JAVA_HOME=/usr/java/jdk1.8.0_45

4. Version verification

Flume-ng version

5.Flume deployment example

Cd / opt/software/flume/bin/

Vi netcat.conf

# Name the components on this agent

A1.sources = R1

A1.sinks = K1

A1.channels = C1

# Describe/configure the source

A1.sources.r1.type = netcat

A1.sources.r1.bind = localhost

A1.sources.r1.port = 44444

# Describe the sink

A1.sinks.k1.type = logger

# Use a channel that buffers events in memory

A1.channels.c1.type = memory

A1.channels.c1.capacity = 1000

A1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel

A1.sources.r1.channels = C1

A1.sinks.k1.channel = C1

6. Run FlumeAgent to listen on port 44444 of this machine

Flume-ng agent-c conf-f netcat.conf-n A1-Dflume.root.logger=INFO,console

[root@hadoop001 bin] # flume-ng agent-c conf-f netcat.conf-n A1-Dflume.root.logger=INFO,console

Info: Including Hadoop libraries found via (/ opt/software/hadoop-2.6.0-cdh6.7.0/bin/hadoop) for HDFS access

Info: Excluding / opt/software/hadoop-2.6.0-cdh6.7.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar from classpath

Info: Excluding / opt/software/hadoop-2.6.0-cdh6.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar from classpath

Info: Including Hive libraries found via (/ opt/software/hive-1.1.0-cdh6.7.0) for Hive access

.

18-03-25 17:52:39 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started

18-03-25 17:52:39 INFO node.Application: Starting Sink k1

18-03-25 17:52:39 INFO node.Application: Starting Source r1

18-03-25 17:52:39 INFO source.NetcatSource: Source starting

18-03-25 17:52:40 INFO source.NetcatSource: Created serverSocket: sun.nio.ch.ServerSocketChannelImpl[ / 127.0.0.1:44444]

7. Open another client

Telnet localhost 44444

[root@hadoop001 ~] # telnet localhost 44444

Trying:: 1...

Telnet: connect to address:: 1: Connection refused

Trying 127.0.0.1...

Connected to localhost.

Escape character is'^]'.

one hundred and eleven

OK18/03/25 18:11:15 WARN source.NetcatSource: Client sent event exceeding the maximum length

18-03-25 18:12:50 INFO sink.LoggerSink: Event: {headers: {} body: 31 31 31 0D 111. }

8. Over.

Thank you for reading this article carefully. I hope the article "how to install and deploy Flume" shared by the editor will be helpful to everyone. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.