Shulou(Shulou.com)05/31 Report--

This article mainly shows you "what are the hadoop problems", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "what are the hadoop problems" this article?

1. Hadoop restart cannot find namenode, probably because no temporary storage directory is specified. By default, it will be stored in the tmp file under the root root directory and will disappear after restart, so you can add temporary file configuration to the core-site.xml file.

2. Hadoop format will ask yes or no,yes uppercase!

3. When configuring SSH, each machine needs to configure its own password, and master and slaves hold their public keys to each other!

4. Hadoop command to operate Java

Bin/hadoop fs-ls

Put the file under the hadoop file, bin/hadoop fs-put.. / input. / in

Delete the file bin/hadoop fs-rmr file, which is not physically deleted, but is placed in the Recycle Bin, where you can set a time threshold to delete beyond the threshold.

Restore and empty bin/hadoop fs-move file

Snapshot:

5, modify the hdfs source code implementation, namenode multi-point, such a namenode downtime will not cause losses! "HDFS High availability"

6, HDFS idea is that hardware equipment is broken is normal, the solution to this is redundancy!

7. Datanode cannot be modified for one write!

8. Heartbeat mechanism: datanode sends regular signals to namenode to tell namenode that I am still alive!

9. Set up hadoop-classpath! Hadoop environment variable!

10. The problem of sharding, sharding can only be very close to block.

Performance tuning

11, combiner preprocessing process, reduce bandwidth, fast transmission speed! Can be optimized here.

12. Deal with small files and synthesize them into large ones.

13. Reduce the output of map phase.

14. Maperd-site.xml sets JVM reuse.

15. If a failure occurs before the maperd task is completed, jobtracker will ask other nodes to re-execute the maperd task.

16. If a failure occurs before the reduce task is completed, jobtracker will ask other nodes to continue with the reduce task.

The above is all the content of this article "what are the hadoop problems?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.