Decryption data partition

There are two types of data partitions, dynamic partitions and static partitions, so how are the two partitions created? How do you use each of them?

I. dynamic Partition

1. Dynamically create a new partition from existing data

2. The partition is automatically created based on the last column value. If the partition does not exist, it will be created.

If the partition exists, it will be overwritten.

Static Partition

1. Static partition example: partition call log by day

Loudacre's customer service phone system generates detailed call logs that analysts use to summarize the previous day's call volume, such as:

Logs are generated on a daily basis, such as:

In the above example, the data is automatically partitioned based on column values. Now we use static partitions

Because the data file does not contain partition data, the partition table is defined in the same way:

2. Load data into a static partition

Using static partitions, you can create new partitions as needed, such as adding a partition for daily call log data:

This command adds the metadata partitioned to the table and creates a subdirectory:

/ user/hive/warehouse/call_logs/call_date=2014-10-02

Then load a day's data into the correct partition

This command moves the HDFS file call-20141002.log to the partition subdirectory

3. Overwrite all data of the partition

This is an introduction to dynamic and static partitions, so how is data partitioned in Impala and Hive? We will continue to share it later. However, there is a threshold for technology, and we should learn more and communicate more in real life, constantly learn from the good experience and knowledge of others, and improve our own knowledge structure. And now big data is still in development, all aspects are not very mature, but also need to continue to pursue in order not to be behind the times, here recommend a Wechat official account "big data cn", not bad, you can follow it when you have time.