Shulou(Shulou.com)05/31 Report--

This article is about how to upgrade hadoop2.7.0 to 2.7.1. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

In the course of use, it is found that the YARN-3832 of hadoop has been resolved in bug,2.6.0, but this bug appears again in 2.7.0.

15-09-28 10:22:34 INFO mapreduce.Job: Task Id: attempt_1443364253801_0310_m_000000_0 Status: FAILEDRename cannot overwrite non empty destination directory / data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60java.io.IOException: Rename cannot overwrite non empty destination directory / data/hadoop/data/tmp/nm-local-dir/usercache/hadoop/filecache/60 at org.apache.hadoop.fs.AbstractFileSystem.renameInternal (AbstractFileSystem.java:735) at org.apache.hadoop.fs.FilterFs.renameInternal (FilterFs.java:236) at org.apache.hadoop .fs.AbstractFilesystem. Rename (AbstractFileSystem.java:678) at org.apache.hadoop.fs.FileContext.rename (FileContext.java:958) at org.apache.hadoop.yarn.util.FSDownload.call (FSDownload.java:366) at org.apache.hadoop.yarn.util.FSDownload.call (FSDownload.java:62) at java.util.concurrent.FutureTask.run (FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter. Call (Executors.java:471) at java.util.concurrent.FutureTask.run (FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615) at java.lang.Thread.run (Thread.java:745)

After looking for it for a long time, it was finally confirmed that the problem was hadoop's bug, not a configuration error. The bug of YARN-3832 was found in the release of 2.7.1 on the official website. So upgrade hadoop.

Upgrade from 2.7.0 to 2.7.1 with the following steps:

1. After downloading the installation package, download the 64-bit package of native to replace native in the new version.

two。 Modify the configuration file in 2.7.1 to keep it consistent with the current cluster configuration, and after modifying the configuration file, distribute it to a temporary directory of each hadoop cluster machine. My list of profiles are: core-site.xml, excludes, hadoop-env.sh, hdfs-site.xml, mapred-env.sh, mapred-site.xml, master, slaves, yarn-env.sh, yarn-site.xml

3. Stop the hadoop cluster.

4. Log in to the server of each hadoop node and back up the installation file directory of hadoop first. Then go to the hadoop installation directory and execute rm-rf bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share

5. Copy the 2.7.1 hadoop file from the temporary directory to the installation directory.

6. Execute start-all.sh, start the hadoop cluster, access http://localhost:50070/dfshealth.html#tab-overview, check whether the hadoop version has been updated, and check whether the status of each node is normal.

Thank you for reading! This is the end of the article on "how to upgrade hadoop2.7.0 to 2.7.1". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.