Shulou(Shulou.com)05/31 Report--

Editor to share with you what the MapReduce in Hadoop is, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

In MapReduce, it is also a master-slave structure, master node: JobTracker, slave node: TaskTracker. There is only one master node and many slave nodes, the master node is on the host, and the slave node is distributed to other machines.

JobTracker:

Function:

1. Responsible for receiving jobs submitted by users

2. Responsible for assigning computing tasks to TaskTracker for execution

3. Monitor the implementation of TaskTracker

TaskTracker:

Function:

1. Execute the computing tasks assigned by JobTracker

JobClient:

Concept: it is the main interface for user jobs to interact with JobTracker.

Function:

1. Responsible for submitting homework and starting

2. Track task execution

3. Access task status and logs, etc.

Communication Protocol based on RPC in Hadoop

1. JobSubmissionProtocol: it is the communication interface between JobClient and JobTracker.

Execution process: TaskTracker sends hearbeat (heartbeat mechanism) to JobTracker at regular intervals to submit the current TaskTracker machine

Memory, CPU, disk and other status information of the device, whether it can receive information such as new tasks, etc. JobTracker received hearbeat

Return to TaskTracker the job or task operation that needs to be done, and whether to start a new task. TaskTracker can be fetched from

Get JobTracker to get the current file system path, need to execute the Jar file path of Job, etc.

2. InterTrackerProtocol: it is the communication interface between TaskTracker and JobTracker.

Function: mainly used to submit, execute Job and obtain current Job status; JobClient can submit Job (submitJob) through this protocol

Get the parameters of the current Job, status, Counter,Task status, file system path, jar file path, and all Job profiles

State, and information about Job queues. In addition, JobClient can also submit tasks such as KillJob and setting Job priorities to JobTracker.

Default settings for drivers in MapReduce:

InputFormat

TextInputFormat

MapperClass

IdentityMapper

MapOutputKeyClass

LongWritable

MapOutputValueClass

Text

PartitionerClass

HashPartitioner

ReduceClass

IdentityReduce

OutputKeyClass

LongWritable

OutputValueClass

Text

OutputFormatClass

TextOutputFormat

Common algorithms in MapReduce

1. Word count

2. Data deduplication

3. Sort

4 、 Top k

5. Select

6. Projection

7. Grouping

8. Multi-table join

9. Single table association

MapReduce principle

Why can MapReduce achieve failure detection?

It is because of the no-sharing framework that all MapReduce are able to achieve failure detection, which means that the tasks are independent of each other.

MapReduce Job:

The MapReduce job (Job) is a unit of work that the client needs to perform: it includes input data, MapReduce programs, configuration information, and so on.

Note: files with file names starting with _ (underscore) cannot be processed in MapReduce.

The above is all the content of the article "what is MapReduce in Hadoop?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.