Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "what is hadoop Scheduler and Compression". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Job scheduling

Scheduler:

FIFO (first in first out) (used in previous versions)

Use FIFO (first-in, first-out) scheduling algorithm to calculate jobs in the order in which jobs are submitted.

Typically, each job uses the entire cluster, so the job must wait until it is its turn to run.

Subsequently, if you set the job priority function, you can set the mapred.job.priority property or

JobClient's setJobPriority () method to set the priority. (very_high,high,normal,low,very_low)

However, in the FIFO algorithm, priority does not support preemption, so high-priority jobs will still be blocked by low-priority jobs.

In hadoopk, MapReduce's scheduler is optional, and the default scheduler is the original queue-based FIFO scheduler

There must also be two multiuser schedulers (Fair [fair] Scheduler and Capacity [competency, status qualification] Scheduler)

Fair Scheduler

The goal of Fair Scheduler is to allow each user to share cluster capabilities fairly. This method is completed in a reasonable time according to the size of the job.

Jobs are placed in the job pool, and by default, each user has its own job pool. You can use the task slots of map and reduce to make

The minimum capacity of the job pool, you can also set the weight of each pool. Fair supports preemption, so if a pool is in a specific period of time

If the resource is not shared fairly, it terminates the task that gets too many resources in the running pool in order to give up the task slot to the pool with insufficient running resources.

Fair scheduler:

Fair scheduling is a method of assigning job resources to all jobs over time.

Can average access to the same shared resources.

1. The Fair Scheduler organizes operations according to resource pools (pool) and distributes resources fairly among these resource pools.

two。 The Fair Scheduler allows minimum shared resources to be assigned to the Resource Pool guarantee (guaranteed).

3. The Fair Scheduler can also limit the number of jobs running concurrently per user and per resource pool.

Installation:

Fair Scheduler is a subsequent module. To use it, you need to put its jar file in the classpath of hadoop.

Method 1: copy fairscheduler.jar to hadoop_home/lib

Method 2: modify the hadoop-classpath in hadoop-env.sh and add the jar package of Fair Scheduler

Finally, you need to set the following properties in mapred-site.xml to let hadoop use the fair scheduler.

Mapred.jobtracker.taskScheduler

Org.apache.hadoop.mapred.FairScheduler

Verification: after restarting the cluster, you can use the http:///scheduler in the web user interface of JobTracker

Check to see if Fair Scheduler is running, and you can see a job scheduler administration page. Just OK.

Capacity Scheduler (capacity scheduler)

The cluster consists of many queues, each of which has an allocation capability. Within each queue, jobs are scheduled according to FIFO mode (priority).

In essence, Capacity Scheduler allows users or organizations to simulate a separate cluster using FIFO for each user or organization.

Compress

Benefits of file compression:

1. You can reduce the disk space needed to store files.

two。 It can speed up the transmission of data on the network and disk.

Compressed format tool algorithm file extension whether to include multiple files can be split java implementation of native implementation

DEFLATE NCMA DEFLATE. Yes or no

Gzip gzip DEFLATE .gz No Yes

Bzip2 bzip2 bzip2 .bz2 No Yes

LZO Lzop LZO .lzo whether or not

This is the end of "what is the hadoop Scheduler and Compression". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.