Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "what is Hive partition and sub-bucket". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Hive partition and bucket ① partition

It means that the data table is divided into multiple regions according to a column or some columns of the data table, which can be understood as a folder in form. For example, if we want to collect log data from a large website, the daily log data of a website is stored on the same table. Because a large number of logs are generated every day, the content of the data table is huge, and it takes a lot of resources to scan the full table when querying. In fact, in this case, we can partition the data table according to the date, and the data on different dates are stored in different partitions. When querying, as long as you specify the value of the partition field, you can find it directly from that partition.

The most common partition is to partition the data by date or hour, and each area is a file, so when we query information, there is no need to scan the whole table, but only need to query the corresponding partition table, which greatly improves the query efficiency.

② split barrel

Sub-bucket is a more fine-grained partition relative to the partition. Separate buckets to distinguish the entire data content according to a column of attributes worthy of hash value. If you want to divide into 3 buckets according to the name attribute, it is to touch 3 pairs of hash values of the name attribute value, and divide the data into buckets according to the modeling results. For example, the data record with a mold result of 0 is stored in a file, the data with module 1 is stored in a file, and the data with module 2 is stored in a file.

This is the end of the content of "what is Hive partition and bucket". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.