Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you about the relevant parameters of permissions and log aggregation in Hadoop YARN. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

Note that before configuring these parameters, you should fully understand the meaning of these parameters in order to prevent the hidden dangers caused by misallocation to the cluster. In addition, these parameters need to be configured in yarn-site.xml.

1. Permission-related configuration parameters

The permissions here are composed of three parts, namely: (1) how administrators and ordinary users distinguish between (2) service-level permissions, such as which users can submit ResourceManager applications to the cluster, and (3) queue-level permissions, such as which users can submit jobs to queue A.

The list of administrators is specified by the parameter yarn.admin.acl.

Service level permissions are achieved by configuring hadoop-policy.xml, which is similar to Hadoop 1.0

Queue-level permissions are configured internally by the corresponding resource scheduler, such as Fair Scheduler or Capacity Scheduler, as detailed in the following sections.

two。 Configuration parameters related to log aggregation

Log aggregation is a centralized log management function provided by YARN, which can upload Container/ task logs to HDFS, thus reducing NodeManager load and providing a centralized storage and analysis mechanism. By default, Container/ task logs exist on each NodeManager, and additional configuration is required if log aggregation is enabled.

(1) yarn.log-aggregation-enable

Parameter explanation: whether log aggregation is enabled.

Default value: false

(2) yarn.log-aggregation.retain-seconds

Parameter explanation: how long the logs aggregated on the HDFS can be kept.

Default value:-1

(3) yarn.log-aggregation.retain-check-interval-seconds

Parameter explanation: how often does it take to check the log and delete it if the condition is met? if it is 0 or a negative number, it is 10 of the previous value.

Default value:-1

(4) yarn.nodemanager.remote-app-log-dir

Parameter explanation: when the application is finished, the log is transferred to the HDFS directory (valid when log aggregation is enabled).

Default value: / tmp/logs

(5) yarn.log-aggregation.retain-seconds

Parameter explanation: remote log directory subdirectory name (valid when log aggregation is enabled).

Default: the log will be transferred to the directory

Under ${yarn.nodemanager.remote-app-log-dir} / ${user} / ${thisParam}

These are the parameters related to permissions and log aggregation in Hadoop YARN shared by the editor. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.