Shulou(Shulou.com)05/31 Report--

Editor to share with you how to calculate the number of reduce in hive, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Hive > select count (1) from serde_regex;Automatically selecting local only mode for queryQuery ID = hadoop_20160125101917_ab5615a4-e6f1-47e3-9e97-6795c3268ceaTotal jobs = 1Launching Job 1 out of 1Number of reduce tasks determined at compile time: 1In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=In order to limit the maximum number of reducers: set hive.exec.reducers.max=In order to set a constant number of reducers: set mapreduce.job.reduces=

Formula for calculation: number of reduce = InputFileSize / bytes per reducer

There are three parameters

Hive.exec.reducers.bytes.per.reducer controls how much input a reducer can handle. Default 256M

Hive > set hive.exec.reducers.bytes.per.reducer;hive.exec.reducers.bytes.per.reducer=256000000

Hive.exec.reducers.max controls the maximum number of reducer. If input / bytes per reduce > max starts the value set by this parameter

Hive > set hive.exec.reducers.max;hive.exec.reducers.max=1009

If the parameter mapreduce.reduce.tasks is specified, it will not be calculated. It will be used for calculation.

Hive > set mapreduce.reduce.tasks;mapred.reduce.tasks=-1 above are all the contents of the article "how to calculate the number of reduce in hive". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.