Shulou(Shulou.com)06/02 Report--

This article mainly introduces the example analysis of HadoopOnDemand configuration items, which is very detailed and has a certain reference value. Friends who are interested must read it!

Configuration items of ringmaster in 3.4HadoopOnDemand

Work-dirs: this is a comma-separated list of paths that will be generated and passed to Hadoop as HOD, the root of the directory where DFS and Map/Reduce data are stored. For example, this is the path where DFS blocks are stored. In general, you specify as many paths as there are disks to ensure that all disks are utilized. The limitations and considerations of the temp-dir variable also apply here.

The number of times the max-master-failures:hadoop master daemon can fail before startup, after which HOD will cause the cluster allocation to fail. In a HOD cluster, sometimes there are one or more "bad" nodes due to some problems, such as the machine does not have java installed, Hadoop is not installed, or the Hadoop version is incorrect. When this configuration item is set to a positive integer, RingMaster will return an error to the client only if hadoopmatser (JobTracker or NameNode) is on the above bad node and the number of startup failures for various reasons mentioned above exceeds the set value. If the number of attempts to start does not exceed the set value, the same hadoopmaster will be assigned to the HodRing when the next HodRing request runs a command. In this way, HOD will try its best to make this allocation successful even if there are some bad nodes in the cluster.

Configuration items of gridservice-hdfs in 3.5HadoopOnDemand

External: if set to false,HOD, you must create your own HDFS cluster on the nodes assigned by the allocate command. Note that in this case, if the cluster is recycled, the HDFS cluster will stop and all data will be lost. If set to true, it will attempt to link to an external configured HDFS system. In general, because the input of the job needs to be placed on the HDFS before the job runs, and the output of the job needs to be persisted, an internal HDFS cluster does not make much sense in a production environment.

Host: the host name of the externally configured NameNode.

The port that the fs_port:NameNodeRPC service binds.

The port that the info_port:NameNodewebUI service binds.

Pkgs: installation directory with bin/hadoop executable files under it. Can be used to use the pre-installed version of Hadoop on the cluster.

Server-params: a comma-separated list of hadoop configuration parameters, each in the form of key-value pairs. These will be used to generate hadoop-site.xml files used by NameNode and DataNode.

Final-server-params: same as above except that it will be marked as final.

Configuration items of gridservice-mapred in 3.6HadoopOnDemand

External: if set to false,HOD, you must create your own Map/Reduce cluster on the nodes assigned by the allocate command. If set to true, it will attempt to link to an external configured Map/Reduce system.

Host: the host name of the externally configured JobTracker.

The port that the tracker_port:JobTrackerRPC service binds.

The port that the info_port:JobTrackerwebUI service binds.

Pkgs: installation directory with bin/hadoop executable files under it.

Server-params: a comma-separated list of hadoop configuration parameters, each in the form of key-value pairs. These will be used to generate hadoop-site.xml files used by JobTracker and TaskTracker.

Final-server-params: same as above except that it will be marked as final.

Configuration items of hodring in 3.7HadoopOnDemand

The directory on the mapred-system-dir-root:DFS where HOD creates a subdirectory and passes the full path to the Hadoop daemon as the value of the parameter 'mapred.system.dir'. The format of the full path is value-of-this-option/userid/mapredsystem/cluster-id. Note that if HDFS has permissions enabled, all users should be allowed to create subdirectories under the path specified here. Setting the value of this configuration item to / user causes HOD to use the user's home directory to generate the value of mapred.system.dir.

Log-destination-uri: a URL that reflects an external static DFS or a path on the local file system on a cluster node. When the cluster is reclaimed, HOD uploads the Hadoop log to this path. To specify the DFS path, use the 'hdfs://path' format. To specify the local file system path on a cluster node, use the 'file://path' format. When HOD reclaims the cluster, the hadoop log is deleted as part of the HOD cleanup process. To persist these logs, you can use this configuration item. The format of the path will be values-of-this-option/userid/hod-logs/cluster-id. Note that all users should be able to create subdirectories under the directory specified here. Setting this value to hdfs://user causes these logs to be transferred to the user's home directory on DFS.

Pkgs: installation directory with bin/hadoop executable files under it. If a HDFSURL,HOD is specified for log-destination-uri to upload logs, this configuration item will be used. Note that this configuration item comes in handy when the user is using a different version of tarball than the external static HDFS.

The above is all the content of the article "sample Analysis of HadoopOnDemand configuration items". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.