Shulou(Shulou.com)06/03 Report--

Slots is an important concept of Hadoop. However, in the related papers of Hadoop, the exposition of slots is difficult to understand. There is little introduction to the concept of slots on the Internet, and for an experienced Hadoop developer, they may have understood the true meaning of slots in their minds, but it is difficult to express it clearly. Hadoop beginners are still confused. Let me try to explain it, with a view to throwing a brick to attract jade.

First of all, slot is not the Core of CPU, nor is it memory chip, it is a logical concept. The number of slot of a node is used to represent the capacity or capacity of a node's resources, so slot is the resource unit of Hadoop.

Hadoop uses slots to manage the resources of the allocation node. Each Job application resource is in slots units, and each node determines its own computing power and memory determines the total amount of slots it contains. When a Job is about to start execution, first apply to JobTracker for slots,JobTracker to allocate free slots,Job and then occupy the slots,Job, and return the slots. Each TaskTracker regularly (for example, Taobao Hadoop heartbeat cycle is 5s) communicates with Jobtracker through heartbeat (hearbeat). On the one hand, it reports its current work status, and JobTracker has to report whether a certain TaskTracker is Alive; or not and report the number of free slots. JobTracker makes use of certain scheduling rules, such as Hadoop default scheduler FIFO or Capacity Scheduler, FairScheduler, and so on. (note: Taobao Hadoop uses the ladder scheduler YuntiScheuler, which is modified based on Fair Scheduler.)

There are two kinds of slots,map slots and reduce slots,map task in Hadoop that use map slots, one by one, and reduce task uses reduce slots. Note: now there is a growing view that the boundary between map slots and reduce slots should be broken and should be regarded as a unified resource pool, they are all resource, so as to improve the utilization of resources. Distinguishing between map slots and reduce slots can easily lead to a shortage of one kind of resource while another is free. In MapR, the next generation framework of Hadoop, the concepts of map slots and reduce slots have been cancelled, and the functions of Jobtracker are divided into two. ResourceManager is used to manage node resources, and ApplicationMaster is used to monitor and schedule jobs. ApplicationMaster is that each Application has a separate instance, and application is a set of tasks submitted by the user, which can be composed of one or more job tasks.

Usually, each tasktracker in a Hadoop contains multiple slots,Job and a task corresponds to a slot in the tasktracker. The formula for calculating the total number of map slots and total reducer slots in the system is as follows:

Total Map slots = number of cluster nodes × mapred.tasktracker.map.tasks.maximum

Total Reducer slots = number of cluster nodes × mapred.tasktracker.reduce.tasks.maximum

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.