Shulou(Shulou.com)06/01 Report--

This article mainly introduces the application of ORC in Hive, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let Xiaobian take you to understand it.

ORC files are not only a column file storage format, but most importantly, they have a high compression ratio and are Split for MapReduce. Therefore, using ORC as the file storage format of the table in Hive can not only greatly save HDFS storage resources, but also greatly improve the query and processing performance of data, because the compression ratio of ORC is higher than other file formats, the amount of input data of the query task is reduced, and the use of Task is also reduced.

Another aspect of performance improvement is to determine whether a file meets the filtering criteria in the WHERE clause by establishing a lightweight index for each field in the ORC file. For example: when executing the HQL statement "SELECT COUNT (1) FROM xxx WHERE adep1", first read the index information from the metadata of the ORC file, quickly locate the offsets where the id=0 is located, and skip the file directly if the information of the id=0 is not found in the index information.

Execute the following statement in Hive, create a new hive table xxx, and specify the storage format as ORC.

CREATE TABLE xxx (name STRING, color STRING) STORED AS ORC

If a table is not stored in ORC format and you now want to change it to ORC format, you can execute the following statement:

Alter table xxx set fileformat orc

Starting with Hive version 0.14, you can execute the following statement to merge ORC small files. The merge occurs at the stripe level, so recompression does not occur.

Alter table xxx [PARTITION partition_spec] CONCATENATE

If you want to view the information about the ORC file, you can execute the following statement, where path is the hdfs path.

Hive-orcfiledump

Starting with Hive 1.1, if you want to view the data contents of ORC, you can execute the following statement:

Hive-orcfiledump-d Thank you for reading this article carefully. I hope the article "how to use ORC in Hive" shared by the editor will be helpful to everyone. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.