Shulou(Shulou.com)06/03 Report--

Background of YARN production

YARN is unique to Hadoop2.x, so before we introduce YARN, let's take a look at the problems with MapReduce1.x:

The single point failure node is under great pressure and is not easy to expand.

The architecture for MapReduce1.x is as follows:

As you can see, 1.x is also the master-slave structure of Master/Slave, and the performance on the cluster is one JobTracker with multiple TaskTracker.

JobTracker: responsible for resource management and job scheduling

TaskTracker: regularly report the health status, resource usage and job execution of this node to JobTracker. You can also receive commands from JobTracker, such as starting a task or ending a task.

So what are the problems with this architecture:

There is only one JobTracker in the whole cluster, which means that there will be a single point of failure. JobTracker nodes are under great pressure not only to receive requests from clients, but also to receive a large number of requests from TaskTracker nodes. Because JobTracker is a single node, it is easy to become a bottleneck in the cluster, and it is not easy to expand the domain. JobTracker carries too many responsibilities, basically the whole cluster is JobTracker to manage version 1.x of the entire cluster only supports MapReduce jobs. Other assignments such as Spark are not supported.

Since version 1.x does not support jobs from other frameworks, we need to build multiple clusters according to different frameworks. This will lead to low resource utilization and high operation and maintenance costs, because multiple clusters will lead to a complex service environment. As shown below:

As we can see in the figure above, I not only need to build different clusters for different frameworks. And these clusters are not always working. As can be seen in the figure above, Hadoop clusters are idle when Spark is busy, Hadoop clusters are idle when Spark clusters are busy, while MPI clusters are not very busy as a whole. This makes it impossible to use resources efficiently because these different clusters cannot use resources with each other. In addition, we have to operate and maintain these different clusters, and the file system cannot be shared. If you need to transfer the data stored in the HDFS on the Hadoop cluster to the Spark cluster for computing, it will also consume a considerable amount of network IO traffic.

So we want to merge these clusters together so that these different frameworks can run on the same cluster, so that we can solve all kinds of problems. As shown below:

It is precisely because in 1.x, there are a variety of problems that led to the birth of YARN, and YARN can make these different frameworks run on the same cluster and schedule resources for them. Let's take a look at the architecture diagram of Hadoop2.x:

In the figure above, we can see that the lowest layer of the cluster is HDFS, on top of which is the YARN layer, while on the YARN layer are various computing frameworks. Therefore, different computing frameworks can share the data on the same HDFS cluster, enjoy the overall resource scheduling, and then improve the utilization of cluster resources, which is called xxx on YARN.

YARN architecture

YARN Overview:

YARN is a resource management system commonly used in the resource scheduling framework to provide unified resource management and scheduling for upper-level applications.

The YARN architecture diagram, which is also the Master/Slave structure:

From the figure above, we can see that YARN is mainly composed of the following core components:

1. ResourceManager, or RM for short, there is only one RM that provides services in the whole cluster at the same time, which is responsible for the unified management and scheduling of cluster resources. And also need to process client requests, such as submitting jobs or ending jobs, etc. And monitor the NM in the cluster, and once a NM fails, you need to tell AM how to handle the tasks running on that NM.

2. NodeManager, referred to as NM, there will be multiple NM in the whole cluster, which is mainly responsible for the resource management and usage of its own node, as well as reporting the resource usage of this node to RM regularly. Receive and process various commands from RM, such as starting Container. NM also needs to process commands from AM, for example, AM will tell NM how many Container it needs to start to run task.

3. ApplicationMaster, or AM for short, each application corresponds to an AM. For example, MapReduce will correspond to one and Spark will correspond to one. It is mainly responsible for the management of the application, requesting resources (Core, Memory) from the RM for the application, and allocating resources to the internal task. AM needs to communicate with NM to start or stop task. Task runs in Container, so AM also runs in Container.

4. Container, which encapsulates a container of CPU, Memory and other resources, is equivalent to the abstraction of a task's running environment.

5. Client, the client, which can submit the job, query the progress of the job, and end the job.

The address of the official YARN document is as follows:

Https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/YARN.html

YARN execution process

Suppose the client submits a job to ResourceManager, and ResourceManager assigns a Container to the job. So ResourceManager communicates with NodeManager, asking the NodeManager to start a Container. This Container is used to start ApplicationMaster, and after ApplicationMaster starts, it registers with ResourceManager. At this point, the client can query the operation of the job through ResourceManager. Then the ApplicationMaster will apply for the resources needed for the job on the ResourceManager. After the application, it will run the job submitted by the client on the corresponding NodeManager, and then NodeManager will run the task in the startup Container.

As shown below:

Also find two good articles about the YARN execution process:

[picture and text] YARN Workflow Yarn Application running process Analysis YARN Environment Construction

After introducing the basic theory, let's build a pseudo-distributed single-node YARN environment. The hadoop version used is as follows:

Hadoop-2.6.0-cdh6.7.0

The official installation documentation address is as follows:

Https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html

1. Download and extract hadoop-2.6.0-cdh6.7.0. For this step, you can refer to my previous article on building a pseudo-distributed environment for HDFS, so I won't repeat it here.

Ensure that the HDFS is in a normal startup state:

[root@localhost ~] # jps3827 Jps3383 NameNode3500 DataNode3709 SecondaryNameNode [root@localhost ~] #

two。 Edit the mapred-site.xml configuration file by adding the following:

[root@localhost ~] # cd / usr/local/hadoop-2.6.0-cdh6.7.0/etc/hadoop [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/etc/hadoop] # cp mapred-site.xml.template mapred-site.xml # copy template file [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/etc/hadoop] # vim mapred-site.xml # add the following content mapreduce.framework.name yarn

3. Edit the yarn-site.xml configuration file by adding the following:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/etc/hadoop] # vim yarn-site.xml # add the following content yarn.nodemanager.aux-services mapreduce_shuffle

4. Start the ResourceManager process and the NodeManager process:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/etc/hadoop] # cd.. /.. / sbin/ [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/sbin] #. / start-yarn.shstarting yarn daemonsstarting resourcemanager, logging to / usr/local/hadoop-2.6.0-cdh6.7.0/logs/yarn-root-resourcemanager-localhost.outlocalhost: starting nodemanager Logging to / usr/local/hadoop-2.6.0-cdh6.7.0/logs/yarn-root-nodemanager-localhost.out [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/sbin] # jps3984 NodeManager # after successful startup, you can see the addition of NodeManager4947 DataNode5252 Jps5126 SecondaryNameNode3884 ResourceManager # and ResourceManager processes This is normal. 4813 NameNode [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/sbin] # netstat-lntp | grep javatcp 00 0.0.0.0usr/local/hadoop-2.6.0-cdh6.7.0/sbin 50090 0.0.0.0lntp * LISTEN 5126/javatcp 00 127.0.0.1 usr/local/hadoop-2.6.0-cdh6.7.0/sbin 42602 0.0.0.0 * LISTEN 4947/javatcp 00 192.168.77.130 LISTEN 8020 0.0.0.0 LISTEN 4813/javatcp 00 0.0.0.0 LISTEN 50070 0.0.0.0 LISTEN 4947/javatcp 00 0.0.0.0 LISTEN 4947/java tcp6 50075 0.0.0.0 LISTEN 4947/java tcp6 00:: 8040:: * LISTEN 5566/javatcp6 0 0: 8042: * LISTEN 5566/javatcp6 0 0: 8088: * LISTEN 5457/java tcp6 0 0: 13562: * LISTEN 5566/javatcp6 0 0: 8030: * LISTEN 5457/javatcp6 0 0: 8031: * LISTEN 5457/javatcp6 0 0: 8032: * LISTEN 5457/java tcp6 0 0: 48929: * LISTEN 5566/javatcp6 0 0: 8033: * LISTEN 5457/java [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/sbin] #

5. If you access ResourceManager through a browser, the default port is 8088, for example, 192.168.77.130 8088, you will access a page like this:

Error resolution:

From the figure above, you can see that there is an unhealthy node, that is, there is a problem with our single-node environment. Click the number marked in the red box to enter the detailed information page, where you can see the following information:

Then check the log file of yarn: yarn-root-nodemanager-localhost.log, and find the following warnings and exceptions:

Obviously, because the disk space has reached 90%, we need to delete some data that we do not have, or expand the disk space. So a bunch of installation packages were deleted, reducing the disk space to less than 90%:

[root@localhost / usr/local] # df-hFilesystem Size Used Avail Use% Mounted on/dev/mapper/centos-root 19G 14G 4.5G 76% / devtmpfs 3.9G 0 3.9G 0% / devtmpfs 3.9G 03.9G 0% / dev/shmtmpfs 3.9G 8.7m 3.9G 1% / runtmpfs 3.9G 03.9G 0% / sys/fs/cgroup/dev/sdb 50G 14G 34G 29% / kvm_data/dev/sda1 497M 127M 371M 26% / boottmpfs 781M 0781M 0% / run/user/0 [root@localhost / usr/local] #

When you refresh the page again, you can find that the node is normal:

At this point, our yarn environment is complete.

If you need to shut down the process, use the following command:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/sbin] # stop-yarn.sh first learn to submit the MapReduce assignment of PI to YARN for execution

Although we don't have a MapReduce environment, we can use some test examples that come with Hadoop to demonstrate how to submit jobs to YARN for execution. Hadoop puts the example package in the following path, and you can see that there are several jar packages:

[root@localhost ~] # cd / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce/ [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # lshadoop-mapreduce-client-app-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-common-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-core-2.6.0-cdh6.7 .0.jarhadoop-mapreduce-client-hs-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-hs-plugins-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-jobclient-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-jobclient-2.6.0-cdh6.7.0-tests.jarhadoop-mapreduce-client-nativetask-2.6.0-cdh6.7.0.jarhadoop-mapreduce-client-shuffle-2.6.0 -cdh6.7.0.jarhadoop-mapreduce-examples-2.6.0-cdh6.7.0.jarliblib-examplessources [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] #

Here we use the jar package hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar to demonstrate:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hadoop jar hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar pi 2 3

Command description:

Hadoop jar executes the command of a jar package job hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar needs to be executed jar package path pi represents the calculated pi, you can write the other two data at the end of the specified two runs of map, and specify that each map task samples 3 times, the multiplication of the two numbers is the total number of samples.

After running the above command, go to the browser page to view, there will be the following three stages:

1. To receive resources, this stage is the resources needed for ApplicationMaster to apply for a job on ResourceManager:

two。 Run the job, and NodeManager will run the task in the launched Container:

3. Job completion:

The output information of the terminal is as follows:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hadoop jar hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar pi 2 3Number of Maps = 2Samples per Map = 3Wrote input for Map # 0Wrote input for Map # 1Starting Job18/03/27 23:00:01 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0 : 218-03-27 23:00:01 INFO mapreduce.JobSubmitter: number of splits:218/03/27 23:00:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1522162696272_000118/03/27 23:00:02 INFO impl.YarnClientImpl: Submitted application application_1522162696272_000118/03/27 23:00:02 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1522162696272_0001/18/03/27 23:00:02 INFO mapreduce.Job: Running job : job_1522162696272_000118/03/27 23:00:10 INFO mapreduce.Job: Job job_1522162696272_0001 running in uber mode: false18/03/27 23:00:10 INFO mapreduce.Job: map 0 reduce 0 reduce 03 INFO mapreduce.Job 27 23:00:15 INFO mapreduce.Job: map 50% reduce 0 reduce 27 23:00:16 INFO mapreduce.Job: map 100% reduce 0 INFO mapreduce.Job 27 23:00:19 INFO mapreduce.Job: map 100 reduce 100-03-27 23:00: 20 INFO mapreduce.Job: Job job_1522162696272_0001 completed successfully18/03/27 23:00:20 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=50 FILE: Number of bytes written=335298 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=536 HDFS: Number of bytes written=215 HDFS: Number of read operations=11 HDFS: Number of large read operations=0 HDFS: Number of write operations=3 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms) = 7108 Total time spent by all reduces in occupied slots (ms) = 2066 Total time spent by all map tasks (ms) = 7108 Total time spent by all reduce tasks (ms) = 2066 Total vcore-seconds taken by all map tasks=7108 Total vcore-seconds taken by all reduce tasks=2066 Total megabyte -seconds taken by all map tasks=7278592 Total megabyte-seconds taken by all reduce tasks=2115584 Map-Reduce Framework Map input records=2 Map output records=4 Map output bytes=36 Map output materialized bytes=56 Input split bytes=300 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=56 Reduce input records=4 Reduce output records=0 Spilled Records=8 Shuffled Maps = 2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed ( Ms) = 2990 CPU time spent (ms) = 2990 Physical memory (bytes) snapshot=803618816 Virtual memory (bytes) snapshot=8354324480 Total committed heap usage (bytes) = 760217600 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=236 File Output Format Counters Bytes Written=97Job Finished in 19.96 secondsEstimated value of Pi is 4.000000000000000000 [root@localhost / usr / local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] #

The above example calculates a Pi value, and let's demonstrate a more classic example in hadoop: wordcount, which is a classic example of word frequency statistics. First, create a file for testing:

[root@localhost ~] # mkdir / tmp/input [root@localhost ~] # cd / tmp/input/ [root@localhost / tmp/input] # echo "hello word" > file1.txt [root@localhost / tmp/input] # echo "hello hadoop" > file2.txt [root@localhost / tmp/input] # echo "hello mapreduce" > file2.txt [root@localhost / tmp/input] # hdfs dfs-mkdir / wc_ input [root @ localhost / tmp/input] # hdfs dfs-put. / file* / Wc_ input [root @ localhost / tmp/input] # hdfs dfs-ls / wc_inputFound 2 items-rw-r--r-- 1 root supergroup 11 2018-03-27 23:11 / wc_input/file1.txt-rw-r--r-- 1 root supergroup 29 2018-03-27 23:11 / wc_input/ file2.txt [root @ localhost / tmp/input] #

Then execute the following command:

[root@localhost / tmp/input] # cd / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hadoop jar. / hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar wordcount / wc_input / wc_output

Phase information displayed on the yarn page:

The output information of the terminal is as follows:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hadoop jar. / hadoop-mapreduce-examples-2.6.0-cdh6.7.0.jar wordcount / wc_input / wc_output18/03/27 23:12:54 INFO client.RMProxy: Connecting to ResourceManager at / 0.0.0.0:803218/03/27 23:12:55 INFO input.FileInputFormat: Total input paths to process: 218-03-27 23:12:55 INFO mapreduce.JobSubmitter: number of splits:218/03/27 23:12:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1522162696272_000218/03/27 23:12:56 INFO impl.YarnClientImpl: Submitted application application_1522162696272_000218/03/27 23:12:56 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1522162696272_0002/18/03/27 23:12:56 INFO mapreduce.Job: Running job: job_1522162696272_000218/03/27 23 : 13:02 INFO mapreduce.Job: Job job_1522162696272_0002 running in uber mode: false18/03/27 23:13:02 INFO mapreduce.Job: map 0 reduce 0 reduce 0 3 INFO mapreduce.Job 27 23:13:06 INFO mapreduce.Job: map 50% reduce 0 INFO mapreduce.Job 27 23:13:07 INFO mapreduce.Job: map 100% reduce 0 pound 03 reduce 27 23:13:11 INFO mapreduce.Job: map 100 reduce 100-03-27 23:13:12 INFO mapreduce.Job: Job job_1522162696272_ 0002 completed successfully18/03/27 23:13:12 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=70 FILE: Number of bytes written=334375 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=260 HDFS: Number of bytes written=36 HDFS: Number of read operations=9 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=2 Launched reduce tasks=1 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms) = 5822 Total time spent by all reduces in occupied slots (ms) = 1992 Total time spent by all map tasks (ms) = 5822 Total time spent by all reduce tasks (ms) = 1992 Total vcore-seconds taken by all map tasks=5822 Total vcore-seconds taken by all reduce tasks=1992 Total megabyte-seconds taken by all map tasks=5961728 Total Megabyte-seconds taken by all reduce tasks=2039808 Map-Reduce Framework Map input records=3 Map output records=6 Map output bytes=64 Map output materialized bytes=76 Input split bytes=220 Combine input records=6 Combine output records=5 Reduce input groups=4 Reduce shuffle bytes=76 Reduce input records=5 Reduce output records=4 Spilled Records=10 Shuffled Maps = 2 Failed Shuffles=0 Merged Map outputs=2 GC time elapsed (ms) = 157CPU time spent (ms) ) = 2290 Physical memory (bytes) snapshot=800239616 Virtual memory (bytes) snapshot=8352272384 Total committed heap usage (bytes) = 762314752 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=40 File Output Format Counters Bytes Written=36 [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] #

View the output result file:

[root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hdfs dfs-ls / wc_outputFound 2 items-rw-r--r-- 1 root supergroup 0 2018-03-27 23:13 / wc_output/_SUCCESS-rw-r--r-- 1 root supergroup 36 2018-03-27 23:13 / wc_output/part-r-00000 [root@localhost / usr/local/hadoop- 2.6.0-cdh6.7.0/share/hadoop/mapreduce] # hdfs dfs-cat / wc_output/part-r-00000 # the actual output is hadoop 1hello 3mapreduce 1word 1 [root@localhost / usr/local/hadoop-2.6.0-cdh6.7.0/share/hadoop/mapreduce] # in part-r-00000

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.