How to test the hdfs of Hadoop
Editor to share with you how to test the hdfs of Hadoop. I hope you will get something after reading this article. Let's discuss it together.
When testing the hadoop-2.4.1 version of hdfs, an error is always reported:
Unable to load native-hadoop library for your platform
Introduction to the native library (NATIVE LIBRARIES) of HADOOP
Hadoop is developed in the Java language, but there are some requirements and operations that are not suitable for using java, so the concept of local library (Native Libraries) is introduced, through which Hadoop can perform some operations more efficiently.
Currently, in Hadoop, local libraries are applied to the compression of files:
When using both compression methods, Hadoop loads local libraries from the $HADOOP_HOME/lib/native/Linux-* directory by default.
If the load is successful, the output is:
DEBUG util.NativeCodeLoader-Trying to load the custom-built native-hadoop library...
INFO util.NativeCodeLoader-Loaded the native-hadoop library
If the load fails, the output is:
INFO util.NativeCodeLoader-Unable to load native-hadoop library for your platform... Using builtin-java classes where applicable
You can set whether to use local libraries in the configuration file core-site.xml of Hadoop:
Hadoop.native.lib
True
Should native hadoop libraries, if present, be used.
The default configuration of Hadoop is to enable local libraries.
In addition, you can set the location to use the local library in the environment variable:
Export JAVA_LIBRARY_PATH=/path/to/hadoop-native-libs
Sometimes you will find that the native library that comes with Hadoop cannot be used, in which case you need to compile the local library yourself. In the $HADOOP_HOME directory, use the following command:
Ant compile-native
After the compilation is complete, you can find the appropriate file in the $HADOOP_HOME/build/native directory, and then specify the path to the file or move the compiled file to the default directory.
After reading this article, I believe you have a certain understanding of "how to test Hadoop's hdfs". If you want to know more about it, you are welcome to follow the industry information channel. Thank you for reading!
