Shulou(Shulou.com)06/02 Report--

This article shows you what the relevant parameters of Capacity Scheduler are, which are concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Capacity Scheduler is the default resource scheduler in YARN.

In the configuration file of Capacity Scheduler, the configuration name of the parameter Y of queue queueX is yarn.scheduler.capacity.queueX.Y. For simplicity, if we mark it as Y, the parameters that can be configured for each queue are as follows:

1. Parameters related to resource allocation

(1) capacity: the resource capacity of the queue (percentage). When the system is very busy, you should ensure that the capacity of each queue is met, and if there are fewer applications per queue, you can share the remaining resources with other queues. Note that the sum of the capacity of all queues should be less than 100.

(2) maximum-capacity: the upper limit (percentage) of resource usage for the queue. Because of resource sharing, the amount of resources used by a queue may exceed its capacity, and the maximum amount of resources used can be limited by this parameter.

M minimum-user-limit-percent: resource guarantee per user (percentage). There is a limit to the amount of resources available to each user in a queue at any time. When applications with multiple users are running simultaneously in a queue, the amount of resources used by each user fluctuates between a minimum value and a * * value, where the minimum value depends on the number of applications running, and the * * value is determined by minimum-user-limit-percent. For example, suppose minimum-user-limit-percent is 25. When two users submit applications to the queue, the amount of resources available to each user cannot exceed 50%, and if three users submit applications, the amount of resources available per user cannot exceed 33%. If four or more users submit applications, the amount of resources available to each user cannot exceed 25%.

(3) user-limit-factor: the maximum amount of resources (percentage) that each user can use. For example, assuming that the value is 30, the amount of resources used by each user cannot exceed 30% of the queue capacity at any one time.

two。 Parameters related to limiting the number of applications

(1) maximum-applications: the maximum number of applications waiting and running in the cluster or queue, which is a strong limit. Once the number of applications in the cluster exceeds this limit, subsequently submitted applications will be rejected. The default value is 10000. The upper limit of the number of all queues can be set by parameter yarn.scheduler.capacity.maximum-applications (which can be regarded as the default value), while a single queue can be set to its own value by parameter yarn.scheduler.capacity..maximum- applications.

(2) maximum-am-resource-percent: the upper limit of the proportion of resources used to run application ApplicationMaster in the cluster. This parameter is usually used to limit the number of active applications. The parameter type is floating point, and the default is 0.1, which means 10%. The upper limit of the proportion of ApplicationMaster resources for all queues can be determined by the parameter yarn.scheduler.capacity. Maximum-am-resource-percent setting (can be seen as the default), while a single queue can be set through the parameter yarn.scheduler.capacity.. Maximum-am-resource-percent sets a value that suits you.

3. Queue access and access control parameters

(1) state: the queue status can be STOPPED or RUNNING. If a queue is in STOPPED state, the user cannot submit the application to the queue or its subqueues. Similarly, if the ROOT queue is in the STOPPED state, the user cannot submit the application to the cluster, but the running application can still run and end normally, so that the queue can exit gracefully.

(2) acl_submit_applications: define which Linux users / user groups can submit applications to a given queue. It is important to note that this property is inherited, that is, if a user can submit an application to a queue, it can submit the application to all of its subqueues. When you configure this property, you use "," to split between users or user groups, and spaces between users and user groups, such as "user1, user2 group1,group2".

(3) acl_administer_queue: assign an administrator to the queue who can control all the applications of the queue, such as killing any application, etc. Again, this property is inherited, and if a user can submit an application to a queue, it can submit the application to all of its subqueues.

An example of a configuration file is as follows:

Yarn.scheduler.capacity.maximum-applications 10000 the maximum number of applications that can be waiting and running at the same time yarn.scheduler.capacity.maximum-am-resource-percent 10000 the maximum number of resources available to run application master in the cluster, which is typically used to limit the number of applications running concurrently. All subqueues of the yarn.scheduler.capacity.root.queues default root queue In this example, there is only one yarn.scheduler.capacity.root.default.capacity 100 default queue's resource capacity yarn.scheduler.capacity.root.default.user-limit-factor 1 the resources available to each user limit the resources available to the yarn.scheduler.capacity.root.default.maximum-capacity 100 Default queue. The status of the yarn.scheduler.capacity.root.default.state RUNNING Default queue, which can be RUNNING or STOPPED. Yarn.scheduler.capacity.root.default.acl_submit_applications * restricts which users can submit applications to the default queue. Yarn.scheduler.capacity.root.default.acl_administer_queue * limits which users can manage applications in the default queue. "*" indicates the maximum number of scheduling opportunities skipped by any user before the yarn.scheduler.capacity.node-locality-delay-1 scheduler attempts to schedule a rack-local container. In general, this value is set to the number of racks in the cluster. By default, it is-1. Indicates that the feature is not enabled. What are the parameters related to Capacity Scheduler? have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.