Shulou(Shulou.com)05/31 Report--

This article mainly introduces "what is the function of flume". In daily operation, I believe that many people have doubts about the function of flume. The editor consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the doubts about "what is the function of flume?" Next, please follow the editor to study!

Flume has its own build hdfs sink for writing hdfs, which has good function and performance, but some shortcomings are difficult to overcome.

1. The collected logs have been writing hadoop, although you can make a rule to write hadoop intermittently, such as setting batchSize, etc., but under great pressure, it can be considered to write hafs almost all the time.

2. Poor fault tolerance. If there are errors in hadoop (for example, hdfs dropping blocks) in the process of log collection, it will crash.

3, and 4, 4, 5, and so on, but hdfs sink has some problems and won't write it.

Write here to collect logs locally, form a file, and then upload the file to the remote hadoop.

It's good to do this.

1, the log is collected to the local file. Hadoop errors, exceptions, and other files that occur in the process of collecting logs do not exist in the project that collects logs into files.

2. Upload to hadoop by means of files, and use hadoop's own fs API, which can be very efficient.

3. A good fault-tolerant mechanism. There is a hadoop problem in the project of uploading files, which leads to file upload failure. It doesn't matter. Just upload the next task and delete the local file only if the upload is successful.

4. It's much better. I won't write it here.

General architecture

Here is the explanation for making false orders:

LocalToHDFSSink.java is the startup class of flume sink, which should be configured in the configuration file, a1.sinks.k1.type = com.fone.flume.sink.localFile.LocalToHDFSSink

The code is as follows:

@ Overridepublic void start () {. CronTriggerFileHDFS cronTriggerFileHDFS = new CronTriggerFileHDFS (); LOG.info ("timer setting, cron expression: {}.", cronExpression); try {cronTriggerFileHDFS.run (filePath, hdfsPath, cronExpression, isKeep);} catch (Exception e) {LOG.warn ("timer error writing to hdfs, error: {}.", e);} sinkCounter.start () .} @ Overridepublic void stop () {.if (cronTriggerFileHDFS! = null) {try {cronTriggerFileHDFS.shutdown ();} catch (Exception e) {/ / TODO Auto-generated catch block e.printStackTrace ();}}.}

Switch the timer.

Timer setting: a1.sinks.k1.local.cronExpression = 0 * / 15 *? Set up in accordance with the requirements of the quartZ settings, those who do not understand to see the quartZ cronTigger settings.

LocalToHDFSSink's file storage is like this: local files are given an initial directory a1.sinks.k1.local.directory, logs are stored in this initial directory, and the dynamic directory structure is set by a1.sinks.k1.local.middleDir.

The remote hadoop gives an initial directory a1.sinks.k1.hdfs.directory, and the other directory structure and files are exactly the same as the local settings, that is, all the contents of the a1.sinks.k1.local.directory directory are copied to a1.sinks.k1.hdfs.directory to complete the work.

The log collection file is generated locally, and when it is not completed, it is suffixed with the file name .tmp, and when it is finished, the .tmp is removed as a sign of whether to copy to hadoop now.

For the rest, look at the code.

At this point, the study on "what is the function of flume" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.