Shulou(Shulou.com)05/31 Report--

This article mainly introduces hadoop streaming how to achieve multi-output expansion, the article introduces in great detail, has a certain reference value, interested friends must read it!

There are two function points implemented by PrefixMultipleOutputFormat

Enter to a different directory according to the prefix of key

Delete the tab in the final output

# # usage # output to different directories according to the prefix of key

$maserati_hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar-libjars. / adts.jar-D mapred.job.name=$name\-D mapred.reduce.tasks=5\-inputformat org.apache.hadoop.mapred.TextInputFormat\-outputformat com.sogou.adt.adts.PrefixMultipleOutputFormat\-input $input\-output $output\-mapper. / m_mapper.sh\-reducer. / M_reducer.sh\-file m_mapper.sh\-file m_reducer.sh

Where outputformat specifies the class of its own time-libjars. / adts.jar imports its own jar package

# mapper and reduer.sh # # m_maper.sh## #! / bin/bash awk-F "'{for

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.