Shulou(Shulou.com)06/03 Report--

Resource manager HA is a feature introduced by hadoop since 2.4. it provides redundancy in the form of Active/Standby in order to eliminate the risk of single point of failure.

1. Overall structure:

2. Failover: there are automatic and manual forms.

Manual: if you switch manually, use the yarn haadmin command to first convert the Active node to standby, and then the standby node to active.

Automatic: RM has a zookeeper-based node election mechanism to determine which node is the active node. You don't need to deploy a zkfc daemon like HDFS, because RM has such functionality embedded in it.

After doing the HA of rm, all nodes and clients list all RM nodes, and the connection will be traversed by polling until a node of active is found. If the active node is down, they will continue to poll. This action is implemented as an org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider class. You can override this behavioral logic by reimplementing the class and specifying a new class name in the yarn.client.failover-proxy-provider configuration item.

3. Configuration

Configuration PropertiesDescriptionyarn.resourcemanager.zk-addressAddress of the ZK-quorum. Used both for the state-store and embedded leader-election.yarn.resourcemanager.ha.enabledEnable RM HA.yarn.resourcemanager.ha.rm-idsList of logical IDs for the RMs. E.g., "rm1,rm2" .yar.resourcemanager.hostname.rm-idFor each rm-id, specify the hostname the RM corresponds to. Alternately, one could set each of the RM's service addresses.yarn.resourcemanager.address.rm-idFor each rm-id, specify host:port for clients to submit jobs. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.scheduler.address.rm-idFor each rm-id, specify scheduler host:port for ApplicationMasters to obtain resources. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.resource-tracker.address.rm-idFor each rm-id, specify host:port for NodeManagers to connect. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.admin.address.rm-idFor each rm-id, specify host:port for administrative commands. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.webapp.address.rm-idFor each rm-id, specify host:port of the RM web application corresponds to. You do not need this if you set yarn.http.policy to HTTPS_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.webapp.https.address.rm-idFor each rm-id, specify host:port of the RM https web application corresponds to. You do not need this if you set yarn.http.policy to HTTP_ONLY. If set, overrides the hostname set in yarn.resourcemanager.hostname.rm-id.yarn.resourcemanager.ha.idIdentifies the RM in the ensemble. This is optional; however, if set, admins have to ensure that all the RMs have their own IDs in the config.yarn.resourcemanager.ha.automatic-failover.enabledEnable automatic failover; By default, it is enabled only when HA is enabled.yarn.resourcemanager.ha.automatic-failover.embeddedUse embedded leader-elector to pick the Active RM, when automatic failover is enabled. By default, it is enabled only when HA is enabled.yarn.resourcemanager.cluster-idIdentifies the cluster. Used by the elector to ensure an RM doesn't take over as Active for another cluster.yarn.client.failover-proxy-providerThe class to be used by Clients AMs and NMs to failover to the Active RM.yarn.client.failover-max-attemptsThe max number of times FailoverProxyProvider should attempt failover.yarn.client.failover-sleep-base-msThe sleep base (in milliseconds) to be used for calculating the exponential delay between failovers.yarn.client.failover-sleep-max-msThe maximum sleep time (in milliseconds) between failovers.yarn.client.failover-retriesThe number of retries per attempt to connect to a ResourceManager.yarn.client.failover-retries-on-socket-timeoutsThe number of retries per attempt to connect to a ResourceManager on socket timeouts.

4. Example (minimum configuration)

Yarn.resourcemanager.ha.enabled

True

Yarn.resourcemanager.cluster-id

Cluster1

Yarn.resourcemanager.ha.rm-ids

Rm1,rm2

Yarn.resourcemanager.hostname.rm1

Master1

Yarn.resourcemanager.hostname.rm2

Master2

Yarn.resourcemanager.webapp.address.rm1

Master1:8088

Yarn.resourcemanager.webapp.address.rm2

Master2:8088

Yarn.resourcemanager.zk-address

Zk1:2181,zk2:2181,zk3:2181

5. Management commands

View node status:

$yarn rmadmin-getServiceState rm1

Active

$yarn rmadmin-getServiceState rm2

Standby

Failover:

$yarn rmadmin-transitionToStandby rm1

$yarn rmadmin-transitionToActive rm2

Note:

When automatic failover is turned on, the system refuses to manage the HA state artificially in order to prevent brain fissure or other inconsistent states. If you are very clear about your behavior, you can specify the-forcemanual option in the switch command.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.