Shulou(Shulou.com)06/02 Report--

This issue of the content of the editor will bring you about 0588-6.1.0-command action specified MapReduce operation parameters invalid problem analysis, the article is rich in content and professional analysis and description for you, after reading this article, I hope you can get something.

one

Purpose of document writing

After completing the development of the MapReduce program locally, package and submit it to the server, then run it with the hadoop jar command on the command line, and dynamically specify the parameters at run time (such as the inner parameters of Map and Reduce, resource pool, etc.). Specify by adding "- D mapreduce.job.queuename= resource pool name" to the command line. This article Fayson mainly deals with the analysis of the invalid parameters of dynamically specified MapReduce jobs.

two

Problem description

The resource pool placement rule on Cloudera Manager is as follows: if the pool specified in the first rule does not exist, it will be created automatically.

From the command line, execute the following command to specify the job under the root.test resource pool:

Hadoop jar xxx.jar-D mapreduce.job.queuename=root.test

As you can see from the figure above, the MapReduce job is not running in the specified resource pool.

three

Analysis of problems

Under the same user, the jar that comes with Hadoop is executed, and the resource pool is dynamically specified in the same way. It is found that the task runs under the specified resource pool.

After the above tests, it is speculated that it may be the problem of the MapReduce code developed by ourselves. by comparing the sample code that comes with Hadoop, it is found that the sample code of Hadoop inherits the Configured class and implements the Tool interface. The MapReduce code I wrote is the following class that does not inherit and implement Hadoop.

four

Problem solving

Add inheritance of the Configured class to the MapReduce code and implement the Tool interface. The modified code is as follows:

After modifying the startup class, repackage the program, and specify the following parameters at runtime:

The discovery task ran successfully under the specified resource pool:

Description of the ToolRunner interface:

Why do dynamic parameters take effect after implementing Tool? When it comes to Tool, you have to mention a class GenericOptionsParser. This class is the basic class for parsing command-line arguments in the Hadoop framework. It can parse command line parameters, so that the program runtime can dynamically specify the configuration of some resources. As you can see in the above code, the run method of the ToolRunner class is called in the main method. The run method in the ToolRunner class uses the GenericOptionsParser class to parse the command-line arguments, and in the end, the run method of the ToolRunner class calls the run method overridden in the MrDriver class. See the following source code:

In the parameters of the ToolRunner.run method, we can see that the incoming tool parameters are the MrDriver class written by ourselves, so the ToolRunner class finally calls our own rewritten run method, and Set the parsed parameters to the Configuration object after parsing the command line parameters through GenericOptionsParser, and finally call the MrDriver.run method to dynamically specify the running parameters of the job.

five

Summary

1. When writing MapReduce programs, you should inherit the Configured class and implement the Tool interface, so that you can specify parameters dynamically when running the jar package, which is much more flexible than hard-coding in the code.

two。 When some parameters need to be modified, there is no need to repackage, compile, deploy, and so on.

3. Not only the parameters related to the resource pool are specified, after the implementation of Tool, other parameters in the configuration file can also be dynamically specified on the command line, but it must be noted that the configuration parameters cannot be written wrong or they will not take effect.

The above is the analysis of the invalid running parameters of the 0588-6.1.0-command action specified by the editor for you. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.