Shulou(Shulou.com)06/02 Report--

This article is to share with you about the characteristics of MapReduce, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Counter

Function: counter is one of the effective means to collect job statistics, which is used for quality control or application-level statistics. The counter can also assist in the diagnosis of system faults.

Task counter

File system counter

FileInputFormat counter

FileOutputFormat counter

Job counter

Job counters are maintained by application master, so there is no need to transfer data between networks

User-defined java counters

User-defined Streaming counters

Sort

Partial sorting

Many applications do not require the global order of the files to be processed.

Full sort

Idea: first, create a series of sorted files, then concatenate these files and finally generate a globally sorted file. The main idea is to use a partitioner to describe the global ordering of the output

Connect

MapReduce can perform "join" operations between large datasets

Map end connection

Reduce end connections are more commonly used

Edge data distribution

Is the additional read-only data required by the job and has been assisted in the processing of the master data set

Use JobConf to configure jobs

Distributed cache

These are the characteristics of MapReduce, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.