Shulou(Shulou.com)05/31 Report--

This article mainly introduces how the HIVE architecture is, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

* Note: this article is based on hive-0.8.1

Architecture diagram:

1. Service

1. Hive has three kinds of service, which are cli,hiveserver and hwi.

2. Cli is a command line tool and the default service, starting with $HIVE_HOME/bin/hive or $HIVE_HOME/bin/hive-- service cli.

3. Hiverserver provides services through thrift. The default port is 10000, and the startup method is $HIVE_HOME/bin/hive--service hiveserver.

4. Hwi is the web interface, and you can access hive through the browser. The default port is 9999, and the startup method is $HIVE_HOME/bin/hive--service hwi.

5. Each service is independent of each other and has its own configuration file (configuration metasotre/namenode/jobtracker, etc.). If the configuration of metasotre is the same, it physically corresponds to the same hive library.

6. Driver is used to interpret, compile, optimize and execute HQL. The Driver of each service is independent of each other.

7. CLI provides users with a command line interface, and each CLI has its own configuration, that is, modifying the configuration in one CLI does not affect other CLI.

8. Multiple JDBC can be connected to the same HiveServer at the same time, and all sessions share a configuration. (note: since 0.9.0, the hiveserver configuration has been reduced from global to session, that is, the configuration of each session is independent of each other, see [HIVE-2503])

9. Multiple browsers can be connected to the same HWI at the same time, and all sessions share a configuration.

II. MetaStore

The MetaStore location is specified by the parameter javax.jdo.option.ConnectionURL and can be freely modified in the session. Relevant parameters include:

Javax.jdo.option.ConnectionDriverName# database driver

Javax.jdo.option.ConnectionURL # database ip port library name, etc.

Javax.jdo.option.ConnectionUserName # user name

Javax.jdo.option.ConnectionPassword # password

By modifying these parameters, you can hot switch between multiple MetaStore, which can be used in HA.

III. NameNode and JobTracker

NameNode is specified by fs.default.name and JobTracker is specified by mapred.job.tracker, both of which can be freely modified in the session to point to different NameNode and JobTracker.

There can be a variety of combinations with MetaStore, for example, to store table1 data in HDFS1 in the same MetaStore, to calculate with JobTracker1, to store table2 data in HDFS2, to calculate with JobTracker2, or to have both tables calculated on JobTracker3.

IV. matters needing attention

1. NameNode and JobTracker are best directed to the same cluster, otherwise data need to be replicated across clusters when computing.

2. What is stored in MetaStore is the absolute path of the table data file, so be careful that it is no longer in the same cluster as NameNode/JobTracker, which leads to cluster replication.

3. Changes to hiveserver and hwi configuration will affect all sessions on the same service. (note: since 0.9.0, hiveserver configuration has been reduced from global to session, that is, the configuration of each session is independent of each other.

Thank you for reading this article carefully. I hope the article "what the HIVE Architecture is like" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.