Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to achieve Hive dynamic partition with Partition function". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "how to use Partition function to achieve Hive dynamic partition".

1. Some time ago, due to the requirement of importing tables (finding data from a table and partitioning overwrite by date to the specified partition table), we studied automatic partitioning in Hive.

Steps:

1. Set up the required partition table

2. Set partition parameters

Set hive.exec.dynamic.partition=true; (can be seen through this statement: set hive.exec.dynamic.partition;) set hive.exec.dynamic.partition.mode=nonstrict; SET hive.exec.max.dynamic.partitions=100000; (if the number of automatic partitions is greater than this parameter, an error will be reported) SET hive.exec.max.dynamic.partitions.pernode=100000

3. Write the code (special note: the date field must be placed at the end), for example:

INSERT OVERWRITE TABLE dw_stage.ds_info_msg partition (cal_dt) select id,..., pv, 0 uv, cal_dtfrom dw_stage.tiny_info_msg

2. If you want to use a function in the partition field, you can also do it in this way; for example:

Set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nostrick;INSERT OVERWRITE TABLE dw_stage.ds_info_msg partition (cal_dt) SELECT id,..., pv, date_sub ('2014-03-23) cal_dtFROM dw_stage.tiny_info_msg07 Thank you for reading, the above is the content of "how to achieve Hive dynamic partition with Partition function". After the study of this article, I believe you have a deeper understanding of how to use Partition function to achieve Hive dynamic partition, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.