Shulou(Shulou.com)06/01 Report--

This article mainly introduces "how hadoop appends new content to HDFS file". In daily operation, I believe many people have doubts about how hadoop appends new content to HDFS file. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful for you to answer the question of "how hadoop adds new content to HDFS file". Next, please follow the editor to study!

In hadoop1.2.1, there are many articles on the Internet saying that this version does not support file append mode, and set the

Dfs.support.append true

Invalid

Since the project I'm working on now requires this feature, it seems impossible to upgrade hadoop, and then look at the source code to find out, where allowBrokenAppend is the identifier used to determine whether the append is supported.

This.allowBrokenAppend = conf.getBoolean ("dfs.support.broken.append", false); if (conf.getBoolean ("dfs.support.append", false)) {LOG.warn ("The dfs.support.append option is in your configuration," + "however append is not supported. This configuration option "+" is no longer required to enable sync ");} LocatedBlock appendFile (String src, String holder, String clientMachine) throws IOException {if (! allowBrokenAppend) {throw new IOException (" Append is not supported. "+" Please see the dfs.support.append configuration parameter ");} startFileInternal (src, null, holder, clientMachine, false, true, false, (short) maxReplication, (long) 0); getEditLog () .logSync ()

As you can see here, it is clear that the dfs.support.append attribute no longer works. If you want to support the append mode, you need to set the following settings

Dfs.support.broken.append true

At this point, the study of "how to add new content to the HDFS file by hadoop" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.