Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to calculate the buffer size needed in the MapReduce process under hadoop". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

In the Map phase, the map function produces intermediate data output and stores it in a memory buffer (the buffer size is specified by the io.sort.mb parameter). Once the occupancy threshold is reached (the default is 80%), the contents of the buffer are written to the local disk, which is known as spill.

Metadata for overflow records (each 16 bytes long) and overflow records are stored in the buffer.

The space allocated to metadata is specified by the parameter io.sort.record.percent, which defaults to 5%, and the rest is allocated to overflow records.

To determine the memory space required for the buffer, you need to calculate the amount of space occupied by overflow records and metadata, respectively.

The specific calculation method is as follows:

Record length = Map output bytes / Map output records = 68022178 / 472293 = 144bytes

Spilled Records Size = Spilled Records * Record length = 144x 472293 = 68022178 = 64m

Metadata Size = Metadata length * Spilled Records = 16 * 472293 = 7556688 = 7m

Io.sort.record.percent = 16 / (16 + 144) = 0.1

Io.sort.mb = Metadata size + Spilled Records size = 64 + 7 = 71m

This is the end of the content of "how to calculate the buffer size needed in the MapReduce process under hadoop". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.