Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "introduction to the type and format of MapReduce". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Type of MapReduce

The input type of the reduce function must match the output type of the map function, but the output type of the reduce function can be different from the input type

Input format

Input slicing and recording

An input fragment is an input block InputSplit interface processed by a single map operation.

Text input

TextInputFormat

Controls the maximum length of a row

KeyValueTextInputFormat

NLineInputFormat

XML

Binary input

SequenceFileInputFormat

SequenceFileAsTextInputFormat

SequenceFileAsBinaryInputFormat

FixedLengthInputFormat

Multiple inputs

MultipleInputs

Database input and output

DBInputFormat is an input format used to read data from a relational database using JDBC. Because it doesn't have any sharing capabilities, you have to be very careful when accessing the database, and running too much mapper reading data in the database may make the database unbearable.

Another method is Sqoop

Output format

OutputFormat

Text output

TextOutputFormat

Key-value pairs are separated by tabs, setting the mapreduce.output.textoutputformat.separator property to change the default delimiter

The corresponding input format is KeyValueTextInputFormat

Binary output

SequenceFileOutputFormat

SequenceFileAsBinaryOutputFormat

MapFileOutputFormat

Multiple output

MultipleOutputFormat

The MultipleOutput class can write data to multiple files

Delayed output

LazyOutputFormat

A subclass of FileOutputFormat produces an output file (part-r-nnnnn), even if the file is empty. Some applications tend not to create empty files

Database output

This is the end of the introduction to the types and formats of MapReduce. Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.