Flume collects logs, writes to hdfs 03/26 Update SLTechnology News&Howtos

Flume collects logs, writes to hdfs

2025-03-26 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Shulou(Shulou.com)06/02 Report--

Install flume first:

It is recommended to maintain a unified user with Hadoop to install Hadoop,flume

This time I use Hadoop users to install flume

Http://douya.blog.51cto.com/6173221/1860390

Start the configuration:

1. Write the configuration file:

Vim flume_hdfs.conf

# Define a memory channel called ch2 on agent1

Agent1.channels.ch2.type = memory

Agent1.channels.ch2.capacity = 10000

Agent1.channels.ch2.transactionCapacity = 100

# agent1.channels.ch2.keep-alive = 30

# define source monitor a file

Agent1.sources.avro-source1.type = exec

Agent1.sources.avro-source1.shell = / bin/bash-c

Agent1.sources.avro-source1.command = tail-n + 0-F / root/logs/appcenter.log

Agent1.sources.avro-source1.channels = ch2

Agent1.sources.avro-source1.threads = 5

Agent1.sources.avro-source1.interceptors = i1

Agent1.sources.avro-source1.interceptors.i1.type = org.apache.flume.interceptor.TimestampInterceptor$Builder

# Define a logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

Agent1.sinks.log-sink1.channel = ch2

Agent1.sinks.log-sink1.type = hdfs

Agent1.sinks.log-sink1.hdfs.path = hdfs://ns1/flume/%y%m%d

Agent1.sinks.log-sink1.hdfs.writeFormat = events-

Agent1.sinks.log-sink1.hdfs.fileType = DataStream

Agent1.sinks.log-sink1.hdfs.rollInterval = 60

Agent1.sinks.log-sink1.hdfs.rollSize = 134217728

Agent1.sinks.log-sink1.hdfs.rollCount = 0

# agent1.sinks.log-sink1.hdfs.batchSize = 100000

# agent1.sinks.log-sink1.hdfs.txnEventMax = 100000

# agent1.sinks.log-sink1.hdfs.callTimeout = 60000

# agent1.sinks.log-sink1.hdfs.appendTimeout = 60000

# Finally, now that we've defined all of our components, tell

# agent1 which ones we want to activate.

Agent1.channels = ch2

Agent1.sources = avro-source1

Agent1.sinks = log-sink1

2, provided that the Hadoop cluster starts normally and is available,

3. Start flume and start collecting logs

Start:

Flume-ng agent-c / usr/local/ELK/apache-flume/conf/-f / usr/local/ELK/apache-flume/conf/flume_hdfs.conf-Dflume.root.logger=INFO,console-n agent1

Error one:

2016-12-06 11 org.apache.flume.sink.hdfs.BucketWriter.open 24 org.apache.flume.sink.hdfs.BucketWriter.open 49036 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO-org.apache.flume.sink.hdfs.BucketWriter.open (BucketWriter.java:234)] Creating hdfs://ns1/flume//FlumeData.1480994688831.tmp

2016-12-06 11 org.apache.flume.sink.hdfs.HDFSEventSink.process 24 org.apache.flume.sink.hdfs.HDFSEventSink.process 49190 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR-org.apache.flume.sink.hdfs.HDFSEventSink.process (HDFSEventSink.java:459)] process failed

Java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName

At org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName (UserGroupInformation.java:366)

At org.apache.hadoop.security.UserGroupInformation. (UserGroupInformation.java:411)

At org.apache.hadoop.fs.FileSystem$Cache$Key. (FileSystem.java:2828)

At org.apache.hadoop.fs.FileSystem$Cache$Key. (FileSystem.java:2818)

At org.apache.hadoop.fs.FileSystem$Cache.get (FileSystem.java:2684)

At org.apache.hadoop.fs.FileSystem.get (FileSystem.java:373)

At org.apache.hadoop.fs.Path.getFileSystem (Path.java:295)

At org.apache.flume.sink.hdfs.BucketWriter$1.call (BucketWriter.java:243)

At org.apache.flume.sink.hdfs.BucketWriter$1.call (BucketWriter.java:235)

At org.apache.flume.sink.hdfs.BucketWriter$9 $1.run (BucketWriter.java:679)

At org.apache.flume.auth.SimpleAuthenticator.execute (SimpleAuthenticator.java:50)

At org.apache.flume.sink.hdfs.BucketWriter$9.call (BucketWriter.java:676)

At java.util.concurrent.FutureTask.run (FutureTask.java:266)

At java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)

At java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)

At java.lang.Thread.run (Thread.java:745)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName

At java.net.URLClassLoader.findClass (URLClassLoader.java:381)

At java.lang.ClassLoader.loadClass (ClassLoader.java:424)

At sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:331)

At java.lang.ClassLoader.loadClass (ClassLoader.java:357)

... 16 more

Resolve:

1,cd / usr/local/hadoop/hadoop/share/hadoop/common/

Cp hadoop-common-2.7.3.jar

2, cd / usr/local/hadoop/hadoop/share/hadoop/common/lib

Hadoop-auth-2.7.3.jar

Just copy the jar package under the lib directory to the lib directory of the flume client.

But there will be a lot of errors, so lazy people copy all the jar packages under lib to lib on the flume side.

Reason: because flume writes data to hdfs, it needs to rely on some hdfs packages

Start:

Flume-ng agent-c / usr/local/ELK/apache-flume/conf/-f / usr/local/ELK/apache-flume/conf/flume_hdfs.conf-Dflume.root.logger=INFO,console-n agent1

Error 2:

2016-12-06 11 org.apache.flume.sink.hdfs.BucketWriter.open 36V 52791 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO-org.apache.flume.sink.hdfs.BucketWriter.open (BucketWriter.java:234)] Creating hdfs://ns1/flume//FlumeData.1480995177465.tmp

2016-12-06 11 org.apache.flume.sink.hdfs.HDFSEventSink.process 36V 52793 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN-org.apache.flume.sink.hdfs.HDFSEventSink.process (HDFSEventSink.java:455)] HDFS IO error

Java.io.IOException: No FileSystem for scheme: hdfs

At org.apache.hadoop.fs.FileSystem.getFileSystemClass (FileSystem.java:2660)

At org.apache.hadoop.fs.FileSystem.createFileSystem (FileSystem.java:2667)

At org.apache.hadoop.fs.FileSystem.access$200 (FileSystem.java:94)

At org.apache.hadoop.fs.FileSystem$Cache.getInternal (FileSystem.java:2703)

At org.apache.hadoop.fs.FileSystem$Cache.get (FileSystem.java:2685)