Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to allocate Container in a Yarn cluster". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

YarnAllocator literally, you should also know that Container is allocated in the Yarn cluster.

Private [yarn] class YarnAllocator (driverUrl: String, driverRef: RpcEndpointRef, conf: YarnConfiguration, sparkConf: SparkConf, amClient: AMRMClient [ContainerRequest], appAttemptId: ApplicationAttemptId, securityMgr: SecurityManager, localResources: Map [String, LocalResource], resolver: SparkRackResolver, clock: Clock = new SystemClock)

Where driverUrl is the address of Driver. When you use YarnAllocator to allocate Container to run Executors, the Driver address that these Executors will contact is the driverRef parameter in the constructor.

The requestTotalExecutorsWithPreferredLocalities method allocates multiple Executor, first saves the allocation request in the queue, and then creates the Executor asynchronously in the daemon thread.

Def requestTotalExecutorsWithPreferredLocalities (requestedTotal: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map [String, Int], nodeBlacklist: Set [String]): Boolean = synchronized {this.numLocalityAwareTasks = localityAwareTasks this.hostToLocalTaskCounts = hostToLocalTaskCount if (requestedTotal! = targetNumExecutors) {logInfo (s "Driver requested a total number of $requestedTotal executor (s).") TargetNumExecutors = requestedTotal allocatorBlacklistTracker.setSchedulerBlacklistedNodes (nodeBlacklist) true} else {false}}

TargetNumExecutors means to indicate how many Executor you want to apply for. The actual creation action is performed in runAllocatedContainers.

LauncherPool.execute () = > {try {new ExecutorRunnable (Some (container), conf, sparkConf, driverUrl, executorId, executorHostname, executorMemory, executorCores, appAttemptId.getApplicationId.toString SecurityMgr, localResources) .run () updateInternalState ()} catch {}

This section is the process of creating an Executor on the assigned Container, wrapped in ExecutorRunner. Where driverUrl is the address of the driver brought in in the constructor.

It can be seen that each driver creates its own separate YarnAllocator.

By the way, many people used to Spark as a cluster, but in fact Spark itself is just a method of calculation, which can be seen as a jar package. Spark's driver will apply for Executor, Yarn or Standalone cluster only when it is running. Standalone cluster refers to Master and Worker, not Spark core, let alone SparkContext.

This is the end of "how to allocate Container in a Yarn cluster". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.