Shulou(Shulou.com)06/01 Report--

Editor to share with you a sample analysis of Hive source code compilation, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Introduction to Hive:

Hive is a data warehouse based on Hadoop, which can map structured data to a table and provide SQL-like functions. Originally provided by Facebook, HQL is used as the query interface, HDFS as the storage layer and MapReduce as the execution layer. It is designed to make SQL skills good, but analysts with weak Java skills can query large amounts of data. In 2008, facebook contributed the Hive project to Apache. Hive provides a relatively complete SQL function (the essence is to convert SQL to MapReduce), and its biggest disadvantage is its slow execution. Hive has its own metadata structure description, which can be stored using relational databases such as MySql\ ProstgreSql\ oracle, but note that all data in Hive is stored in HDFS. Hive has good flexibility and expansibility, supports UDF, self-defined storage format, and is suitable for offline data processing.

Hive architecture:

1. User interface: including CLI, Client, WUI. The most common one is that CLI,CLI is the shell command line, and when Cli starts, it starts a copy of Hive at the same time. Client is the client of Hive, and the user connects to Hive Server. When you start Client mode, you need to indicate the node where Hive Server is located, and start Hive Server on that node. WUI accesses Hive through a browser.

two。 Metadata storage: usually stored in relational databases such as mysql, derby

3. Interpreter, compiler, optimizer, executor: complete HQL query statements from lexical analysis, syntax analysis, compilation, optimization and query plan generation. The generated query plan is stored in HDFS and subsequently executed by a MapReduce call.

The data in 4.Hadoop:Hive is stored by HDFS and calculated by MapReduce.

Hive source code compilation:

We choose the CDH version to compile, and take the hive-1.1.0-cdh6.7.0 version as an example.

Preliminary preparation:

1. Configure the jdk1.7 environment.

two。 Install maven.

3. Install the hadoop-2.6.0-cdh6.7.0 environment (version number corresponds to hive).

Note: I used Apache 2.8version of hadoop in the earlier stage, but it has been changed to CDH version now.

1. Download and decompress the source code package hive-1.1.0-cdh6.7.0-src.tar.gz# download address: http://archive.cloudera.com/cdh6/cdh/5/[root@hadoop000 ~] # su-hadoop [hadoop@hadoop000 ~] $cd source [hadoop@hadoop000 source] $wget http://archive.cloudera.com/cdh6/cdh/5/hive-1.1.0-cdh6.7.0-src.tar.gz[hadoop@hadoop000 source] $tar -xzvf hive-1.1.0-cdh6.7.0-src.tar.gz2. It takes a while to compile [hadoop@hadoop000 source] $cd hive-1.1.0-cdh6.7.0#. [hadoop@hadoop000 hive-1.1.0-cdh6.7.0] $mvn-Phadoop-2-Pdist-DskipTests-Dmaven.javadoc.skip=true clean package. [INFO] Hive ODBC. SUCCESS [5.811 s] [INFO] Hive Shims Aggregator.. SUCCESS [0.755 s] [INFO] Hive TestUtils.. SUCCESS [1.294 s] [INFO] Hive Packaging.. SUCCESS [01:41 min] [INFO]-[INFO] BUILD SUCCESS [INFO]- -[INFO] Total time: 18:22 min [INFO] Finished at: 2018-06-02T11:37:54+08:00 [INFO] Final Memory: 144M/494M [INFO]-- -- # A successful compilation will generate a tar package under the packaging/target directory under the source directory The related hive module jar package can be found in this tar package. File name: apache-hive-1.1.0-cdh6.7.0-bin.tar.gz# can use this package to install Hive, or you can download the corresponding version of the compiled installation package directly from this website http://archive.cloudera.com/cdh6/cdh/5/. The above is all the contents of the article "sample Analysis of Hive Source Code compilation". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.