Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to realize the bucket table in Hive. The editor thinks it is very practical, so I share it for you as a reference. I hope you can get something after reading this article.

Bucket table

In general, when it is difficult to create a partition on a column, we use a bucket, such as a field that is often filtered, which results in a large number of partitions if used as a partition field. In Hive, bucket fields are hashed, providing additional data structures in to improve query efficiency.

Similar to partition tables, bucket tables are organized by dividing files on the HDFS into multiple files. Sub-bucket can speed up data sampling and improve the performance of join (the field of join is a sub-bucket field). Because sub-bucket can ensure that the data corresponding to a key is in a specific bucket (file), skillfully selecting sub-bucket field can greatly improve the performance of join. In general, bucket fields can be selected for fields that are often used in filter operations or join operations.

We can use set.hive.enforce.bucketing = true to enable the bucket setting.

When using a bucket table, it is best to set the bucketmapjoin flag to true. The specific configuration parameters are:

SET hive.optimize.bucketmapjoin = true

CREATE TABLE table_name

PARTITIONED BY (partition1 data_type, partition2 data_type, … (.) CLUSTERED BY (column_name1, column_name2, …)

SORTED BY (column_name [ASC | DESC], …)]

This is the end of INTO num_buckets BUCKETS; 's article on "how to implement bucket-splitting table in Hive". I hope the above content can be helpful to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.