Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the working mechanism of hive". Interested friends may wish to take a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the working mechanism of hive"?

Hive is a data warehouse tool based on Hadoop, which can map structured data files to a database table and provide similar SQL query functions.

The working Mechanism of hive

1. Build a library in hive

-record in hive Metabase

-create a folder with the name "library name .db" under the default path of hdfs / user/hive/warehouse/

2. Create a table in the library of hive

-record in hive Metabase

-create a folder with the "table name" under the default path of hdfs / user/hive/warehouse/ library. db/

3. The difference between internal table and external table in hive.

-when creating a table, the internal table does not need to specify the path where the data is stored, but defaults to / user/hive/warehouse/.

-when an external table creates a table, specify the external keyword and the path where the data is stored (specify where the data to be analyzed is located)

-when the internal table is deleted, the metadata is cleared and the table folder and the data in it are deleted.

-when an external table is deleted, only metadata is cleared

4. The data of hive table can be saved in a variety of file formats, the most common one is textfile, but the sequenceFile format has better performance.

-sequencefile is a binary file

-the content in the file is organized as key:value

-there is an optimized scenario in hadoop where sequencefile can be used

Small files are merged into large files:

-read a small file and append the file name of the small file as key and the content as value to a large sequencefile file

-sequencefile file format supports better compression performance, and hadoop's mapreduce program can read data directly from sequencefile's compressed files.

5. The method of running HQL statement directly in the shell of linux

/ / cli shell

Hive-S-e'select country,count (*) from tab_ext' > / home/hadoop/hivetemp/e.txt

This kind of running mechanism is very important, and it is used in production to organize a large amount of HQL logic into a batch-executed shell script.

6. Partition table

The significance of partition tables is that statistics can be carried out for a partition to reduce the statistical data set.

Create a partition table using the keyword partitioned by (country string)

When importing data into a partition table, you need to specify the partition load data to which the data belongs. Partition (country='china')

Hive will create a partition subfolder in the table directory of hdfs, named country=china, under which the data of this partition will be placed.

Queries and statistics for partitions can be where country='china' as long as the where condition is specified and the partition identity is treated as a normal table field.

At this point, I believe you have a deeper understanding of "what is the working mechanism of hive". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.