Shulou(Shulou.com)06/02 Report--

This article mainly shows you "how to configure HadoopOnDemand", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to configure HadoopOnDemand" this article.

HadoopOnDemand: configuration Guid

1. Brief introduction

This document describes some of the most important and commonly used configuration items for HadoopOnDemand (HOD). These configuration items can be specified in two ways: an INI-style configuration file, and a HODshell command-line option specified through the-- section.option [= value] format. If the same option is specified in both places, the value on the command line overrides the value in the configuration file.

You can get a brief description of all configuration items with the following command:

$hod--verbose-help

two。 Segment

The HOD configuration file is divided into the following configuration segments:

Configuration items for hod:HOD client

Resource_manager: specify the configuration items for the resource manager to use, as well as some other parameters required to use the resource manager.

Configuration items for the ringmaster:RingMaster process

Configuration items for the hodring:HodRing process

Configuration items for the gridservice-mapred:Map/Reduce daemon

Configuration items for the gridservice-hdfs:HDFS daemon

HOD configuration items in 3.HadoopOnDemand

The following section first describes some configuration items common to most HOD configuration segments, and then describes the configuration items specific to each configuration segment.

3.1 General configuration items

Some configuration items are defined in multiple segments in the HOD configuration. Configuration items defined in a segment are used by all processes to which that paragraph applies. These configuration items have the same meaning, but can have different values in different segments.

Temporary directory used by the temp-dir:HOD process. Make sure that the user running hod has permission to create subdirectories under this specified directory. If you want to use a different temporary directory for each allocation, you can use environment variables, which the explorer makes available to the HOD process. For example, when Torque is set, make-- ringmaster.temp-dir=/tmp/hod-temp-dir.$PBS_JOBID causes ringmaster to use a different temporary directory for each application; Troque expands this environment variable before ringmaster starts.

Debug: numeric type, with a range of 1-4. 4 will generate the most log information.

Log-dir: the directory where the log files are stored. The default value is / logs/. The limitations and considerations of the temp-dir variable are also used here.

Xrs-port-range: the range of ports in which one of the available ports is selected to run the XML-RPC service.

Http-port-range: the range of ports in which one of the available ports is selected to run the HTTP service.

Java-home: the location of the Java for Hadoop to use.

The address to be bound by the syslog-address:syslog daemon. The format is host:port. If this option is configured, the HOD log information will be recorded to the syslog at this location.

Configuration items of hod in 3.2HadoopOnDemand

Cluster: the descriptive name of the cluster. For Torque, this value is specified as the 'Nodeproperty' of all nodes in the cluster. HOD uses this value to calculate the number of available nodes.

Client-params: a comma-separated list of hadoop configuration parameters, each of which is a key-value pair. A hadoop-site.xml is generated on the submission node to run the Map/Reduce job.

Job-feasibility-attr: a regular expression that specifies whether and how to check the feasibility of a job-resource manager restrictions or scheduling restrictions. This is currently done through the 'comment' property of the torque job, which is not turned on by default. Once this configuration item is set, HOD uses it to determine which kinds of limits are enabled and whether to reclaim the cluster or remain queued when requests exceed or accumulate beyond the limit. The torquecomment property can be updated periodically by some external mechanism. For example, the comment property is updated by checklimits.sh in the hod/support directory, so that setting the value of job-feasibility-attr is equal to TORQUE_USER_LIMITS_COMMENT_FIELD, and "User-limitsexceeded.Requested: ([0-9] *) Used: ([0-9] *) MaxLimit: ([0-9] *)" causes HOD to behave accordingly.

Configuration items of resouce_manager in 3.3HadoopOnDemand

Queue: the queue name configured in Explorer, where the job will be submitted.

Batch-home: an installation directory under which there are executable files for the resource manager in the 'bin'.

Env-vars: a comma-separated list of key-value pairs in the form of key=value, which is passed to the job running on the compute node. For example, if ptyhon is not installed in a regular location, the user can specify the path to the python executable by setting the environment variable 'HOD_PYTHON_HOME'. After that, the HOD process running on the compute node can use this variable.

The above is all the content of this article "how to configure HadoopOnDemand". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.