Shulou(Shulou.com)06/01 Report--

This article will give you a detailed explanation on how to install and deploy flume1.4. The editor thinks it is very practical, so I share it with you for reference. I hope you can get something after reading this article.

I. introduction

Flume-ng is a distributed, highly reliable and efficient log collection system. Flume-ng means a new version of flume, where "ng" means new generate. Currently, flume-ng 1.4 is the latest version. Compared with flume, flume-ng has undergone great changes, because it has been in the version of flume0.9 and has not been upgraded to flume-ng. Recently, due to the needs of the project, I made an upgrade and found some problems. Please record it and share it with you.

II. Version Notes

Flume-ng 1.4.0

Third, installation steps

Download, extract, install JDK, set up environment variables section already has a lot of introductory questions, not explained. It is important to note that flume-ng does not need to zookeeper, does not need to be set.

IV. Flume-ng bug

Running flume-ng after installation will get an error message, which is mainly due to the problem of the shell script. I will upload the modified flume-ng as follows, with a note: the line under # zhangzl is the part that needs to be modified.

Fifth, test configuration file

Create an example-conf.properties file in the conf directory with the following attributes:

1 # Describe the source 2 a1.sources = R13 a1.sinks = K14 a1.channels = c156 # Describe/configure the source 7 a1.sources.r1.type = avro 8 a1.sources.r1.bind = localhost 9 a1.sources.r1.port = 4444410 11 # Describe the sink12 # output the data to the log 13 a1.sinks.k1.type = logger14 15 16 # Use a channel which buffers events in memory17 a1.channels.c1.type = memory18 a1.channels.c1.capacity = 100019 a1.channels. C1.transactionCapacity = 10020 21 # Bind the source and sink to the channel22 a1.sources.r1.channels = c123 a1.sinks.k1.channel = C1

Run the command

6.1 start the agent

[hadoop@hadoop1 conf] $flume-ng agent-n A1-f example-conf.properties

6.2 start the avro-client client to send data to the agent agent-you need to start a new window separately

[hadoop@hadoop1 conf] $flume-ng avro-client-H localhost-p 44444-F file01

7. Check the results

1 22:26:34 on 14-01-16 INFO ipc.NettyServer: [id: 0x0100c7e4, / 127.0.0.1 OPEN2 54289 = > / 127.0.0.1 OPEN2 22:26:34 INFO ipc.NettyServer: [id: 0x0100c7e4, / 127.0.1 OPEN2 54289 = > / 127.0.0.1 OPEN2 54289: / 127.0.0.1 INFO ipc.NettyServer: / 127.0.0.1 OPEN2 54289: / 127.0.0.1 OPEN2 54289: / 127.0.0.1 INFO ipc.NettyServer: / 127.0.0.1 OPEN2 54289: / 127.0.0.1 INFO ipc.NettyServer: / 127.0.0.1 OPEN2 54289: / 127. / 127.0.0.1 id 54289 = > / 127.0.0.1 id 44444] CONNECTED: / 127.0.0.1 id 542894 14-01-16 22:26:36 0x0100c7e4: / 127.0.0.1 0x0100c7e4 54289: > / 127.0.0.1 DISCONNECTED5 14-01-16 22:26:36 INFO ipc.NettyServer: [CONNECTED: 0x0100c7e4 / 127.0.0.1 UNBOUND6 54289: > / 127.0.0.1 UNBOUND6 22:26:36 on 14-01-16 INFO ipc.NettyServer: [id: 0x0100c7e4 / 127.0.0.1 CLOSED7 22:26:36 on 14-01-16 INFO ipc.NettyServer: Connection to / 127.0.0.1 disconnected.8 54289 disconnected.8 14-01-16 22:26:38 INFO sink.LoggerSink: Event: {headers: {} body: 68 65 6C 6F 20 77 6F 72 6C 64 hello world} this is the end of the article on "how to install and deploy flume1.4" Hope that the above content can be helpful to you, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.