Shulou(Shulou.com)06/01 Report--

This article mainly shows you "what to do when Flume collects data and produces a large number of small files on HDFS", which is easy to understand and clear. I hope it can help you solve your doubts. Let me lead you to study and learn how to generate a large number of small files on HDFS when Flume collects data.

Call the agent configuration file:

# Describe the sink add:

The default value of a1.sinks.k1.hdfs.rollSize=0a1.sinks.k1.hdfs.rollCount=0rollSize is 1024, and when the temporary file reaches this size (in bytes), it scrolls to the target file. If set to 0, the file is not scrolled based on the temporary file size. The default value of rollCount is 10. When the events data reaches this amount, the temporary file is scrolled to the target file. If set to 0, the file is not scrolled according to the events data.

There is another solution that corresponds to a large number of small files, which is to set up a1.sinks.k1.hdfs.minBlockReplicas=1. Because the file will scroll for testing because of the copy of the block in which it is located.

These are all the contents of this article entitled "how to generate a large number of small files on HDFS when Flume collects data?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.