Shulou(Shulou.com)05/31 Report--

This article mainly shows you "where the data of HIVE is stored", which is easy to understand and well organized. I hope it can help you solve your doubts. Let me lead you to study and learn this article "where is the data of HIVE stored?"

First of all, Hive does not have a special data storage format, nor does it index the data. Users are very free to organize tables in Hive, as long as they tell Hive the column delimiter and row separator in the data when creating the table, and Hive can parse the data.

Second, all the data in Hive is stored in HDFS, and Hive contains the following data model: Table,External Table,Partition,Bucket.

1. Table in Hive is similar in concept to Table in database, and each Table has a corresponding directory to store data in Hive. For example, a table xiaojun, whose path in HDFS is / warehouse / xiaojun, where wh is the directory of the data warehouse specified by ${hive.metastore.warehouse.dir} in hive-site.xml, where all Table data (excluding External Table) is stored in this directory.

2. Partition corresponds to a dense index of Partition columns in the database, but the organization of Partition in Hive is very different from that in the database. In Hive, a Partition in a table corresponds to a directory under the table, and all Partition data is stored in the corresponding directory. For example, if the xiaojun table contains two Partition, dt and city, then the HDFS subdirectory of ctry=US corresponds to dt=20100801: / warehouse / xiaojun/dt=20100801/ctry=US; corresponds to dt=20100801, and the HDFS subdirectory of ctry=CA is; / warehouse / xiaojun/dt=20100801/ctry=CA

3. Buckets calculates the hash for the specified column and splits the data according to the hash value, in order to be parallel, each Bucket corresponds to a file. If you divide the user column into 32 bucket, first calculate the hash for the value of the user column. The corresponding HDFS directory with a hash value of 0 is: / warehouse / xiaojun/dt = 20100801 hand ctryhands UShand HDFS directory with a hash value of 20: / warehouse / xiaojun/dt = 20100801/ctry=US/part-00020

4. External Table points to the data that already exists in HDFS, and you can create a Partition. It is the same as Table in the organization of metadata, but the storage of actual data is quite different.

During the process of creating Table and loading data (both of which can be done in the same statement), the actual data will be moved to the data warehouse directory during loading, and then the access to the data pairs will be completed directly in the data warehouse directory. When you delete a table, the data and metadata in the table are deleted at the same time.

External Table has only one process, loading data and creating tables at the same time (CREATE EXTERNAL TABLE... LOCATION), the actual data is stored in the HDFS path specified after the LOCATION and is not moved to the data warehouse directory. When you delete an External Table, delete only

The above is all the contents of the article "where is HIVE's data stored?" Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.