Shulou(Shulou.com)06/01 Report--

This article mainly talks about "what are the pitfalls in the process of Hadoop development". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Now let the editor take you to learn what are the pitfalls in the process of Hadoop development.

1. Firewall reason: check whether the firewall of each node is shut down successfully. (focus on checking NameNode)

2. Check whether the mapping between IP address and host name is successful.

3. Check whether NameNode is in safe mode.

4. Check whether NameNode has been formatted

5. Check whether the configuration of the configuration file is successful

6. Check whether the version number of the namespaceID stored in the NameNode node and the DataNode node is the same

OK, after we have checked the above 6 points, if the problem has not been solved, then we can check the relevant log files.

OK, so far I'm going to introduce you to a few exception problems that you often encounter in the development process:

1. Possible reasons why there is no NameNode when starting hadoop

This problem is often encountered by beginners of Hadoop. There may be three reasons for this problem:

1), NameNode is not formatted (6 checks to include)

First delete the directories corresponding to hadoop.tmp.dir (that is, logs and tmp), and then format NameNode

2) check whether the mapping between IP address and hostname is successful (6 checks to include)

3) check that the configuration of the configuration file is successful (6 checks to include), focusing on hadoop-env.sh, core-site.xml, hdfs-site.xml, mapred-site.xml, and slaves.

2 、 Name node is in safe mode.

For example:

Reason: NameNode will enter the safe mode when it starts, and will exit after a countdown of 30 seconds. You can only view whether you can add, delete or change it in safe mode. However, if the block blocks lost by the data node DataNode reach a certain proportion, the system is always in safe mode, that is, read-only state.

Solution:

1) in the configuration file hdfs-site.xml of HDFS, modify the corresponding value of dfs.safemode.threshold.pct to a smaller value. The default value is 0.999f.

2) execute the command hadoop dfsadmin-safemode leave to force NameNode to leave safe mode. (6 checks to include)

3), could only be replicatied to 0 nodes, instead of 1.

For example:

The possible phenomenon of this exception: the processes displayed by the command jps are normal, but if you view it with the web interface, the live nodes displayed is 0, which means that the data node DataNode does not start normally, but the data node DataNode starts normally again.

The possible causes of this problem are:

1) for firewall reasons, check whether the firewalls of all nodes are shut down successfully. (6 checks to include)

2), disk al reason: execute the command df-disk to check the disk space usage, and adjust the disk space if the disk space is insufficient.

If there is insufficient disk space, perform the following steps to check:

3. If none of the above methods work, you can use the following methods to deal with them (but this method will cause data loss, so use it with caution!)

Delete the directory corresponding to hadoop.tmp.dir, and then reformat the NameNode. (6 checks to include)

4. Start the Times wrong java.net.UnknownHostException

Reason: the hostname in the cluster is not mapped to the corresponding IP address (6 checks to include)

Solution: add the mapping between the hostnames and IP addresses of all nodes in the / etc/hosts file.

5. The TaskTracker process starts, but the DataNode process does not start

Solution: first delete the folder corresponding to hadoop.tmp.dir, and then reformat the NameNode.

6 、 java.lang.OutOfMemoryError

Cause analysis: this exception is obviously due to insufficient memory in JVM. It is necessary to modify the JVM memory size corresponding to the DataNode of all data nodes.

Method: in the configuration file mapred-site.xml of MapReduce, modify the value corresponding to mapred.child.java.opts.

Note: in general, the memory usage of JVM should be half of the total memory size. For example, if the memory size of our server is 4G, it is set to 2048m, but this value may not be the value of * *. Among them

-Xms indicates the size of memory initialization, and-Xmx represents the available memory.

View the hardware information of server memory under linux:

7 、 Incompatible namespaceIDs in

Reason analysis: every time the NameNode is formatted, a new namespaceID will be generated. If you format the NameNode several times, it may cause the version number stored in the NameNode node and the DataNode node to be inconsistent.

Solution:

1) check whether the version number of the namespaceID is the same in the NameNode node and the DataNode node, and if not, change it to the same value, and then restart the node. (6 checks to include)

2) delete the directory corresponding to hadoop.tmp.dir, and then reformat the NameNode. (6 checks to include)

At this point, I believe you have a deeper understanding of "what are the pits in the process of Hadoop development?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.