Shulou(Shulou.com)06/03 Report--

Hive common commands and settings

Alter table ad_app.app_accounting_daily add columns (return_cost bigint) cascade

Data tilt

Set hive.map.aggr=true

Set hive.groupby.skewindata=true

Set mapreduce.job.queuename = root.data.adonline

Set mapred.reduce.tasks=8192

Set mapreduce.job.priority=VERY_HIGH

Set hive.groupby.mapaggr.checkinterval = 100000

Set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

Set mapred.map.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

Set hive.exec.compress.intermediate=true

Set mapred.compress.map.output=true

Set mapred.max.split.size=256000000;-determines the maximum file size processed by each map, in B

Set mapred.min.split.size.per.node=1;-the smallest file size that can be processed in the node

Set mapred.min.split.size.per.rack=1

Control the number of map

Set mapred.max.split.size=128000000

Set mapred.min.split.size.per.node=64000000

Set mapred.min.split.size.per.rack=64000000

Set hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat

Set the maximum number of reducer

Set hive.exec.reducers.max=8192

Control map reduce memory

Set mapreduce.map.memory.mb=4096

Set mapreduce.reduce.memory.mb=4096

Set mapreduce.reduce.java.opts=-Xmx3800M

Set mapred.child.map.java.opts=-Xmx3800M

Set mapreduce.map.java.opts=-Xmx3800M

ALTER TABLE table_name ADD IF NOT EXISTS PARTITION (dt='20130101') LOCATION'/ user/hadoop/warehouse/table_name/dt=20130101'; / / add one partition at a time

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.