Shulou(Shulou.com)06/03 Report--

Lu Chunli's work notes, who said that programmers should not have literary style?

Hive provides three service modes, namely, CLI (command line interface), Hive Web and Hive Client (such as JavaApi).

1. Hive command line mode (CLI)

There are two ways to start Hive command line mode

Bin/hive or bin/hive-- service cli

Hive command options

[hadoop@nnode hive1.2.0] $bin/hive-- helpUsage. / hive-- service serviceName Service List: beeline cli help hiveburninclient hiveserver2 hiveserver hwi jar lineage metastore metatool orcfiledump rcfilecat schemaTool version # here corresponds to the parameter Parameters parsed: # corresponding to hive allows users to specify an attached jar package separated by colons, such as custom extensions. -- auxpath: Auxillary jars # specifies the file directory, overriding the default attribute configuration in $HIVE_HOME/conf-- config: Hive configuration directory # the service that needs to be started, defaults to cli. For more information, please see Service List:-- service: Starts specific service/component. Cli is defaultParameters used: HADOOP_HOME or HADOOP_PREFIX: Hadoop install directory HIVE_OPT: Hive optionsFor help on a particular service: # help querying a specific service name. / hive-- service serviceName-- helpDebug help:. / hive-- debug-- help## uses the version service [hadoop@nnode hive1.2.0] $bin/hive-- service versionHive 1.2.0Subversion git://localhost.localdomain/home/sush/dev/hive.git-r 7f237de447bcd726bb3d0ba332cbb733f39fc02fCompiled by sush on Thu May 14 18:00:25 PDT 2015From source with checksum 03a73b649153ba8e11467a779def6315 [hadoop@nnode hive1.2.0] $# # execute-service does not take any parameter [hadoop@nnode hive1.2.0] $bin/hive-- service is equivalent to bin/hive-- service cli

Service List includes most of the CLI that will be used, and can be started by-- the name of the service name service. The default is to start cli. Note that individual services actually provide a quick startup.

Common services are shown in the following figure:

Hive cli command parameters

[hadoop@nnode hive1.2.0] $bin/hive-- verbose-- helpusage: hive-d _ mai _ me define the parameters used by the hive command line For example, the database specified by-d Atom B or-- define Atom B-- database executes the SQL statement from the command line-f executes the SQL statement in the file-H -- help display help-- hiveconf given parameter values override parameter values in hive-default.xml or hive-site.xml-- hivevar definitions apply to variables in hive For example,-- hivevar axib (equivalent to-d)-I initialized sql file-Sminicare static mode (no output)-v -- verbose verbose mode [hadoop@nnode hive1.2.0] $# such as # hive- e "" # hive- e "> aaa#hive-S-e" > aaa#hive-e 'select a.col from tab1 a' # hive- f hdfs://:/hive-script.sql#hive-I / home/my/hive-init.sql#hive > source file#hive >! ls # use the shell command # Hive > dfs-ls / # use the hdfs dfs command (omit hdfs)

A variable or attribute

In CLI, you can display or modify the value of a variable through the set command, or you can specify it through the command space.

Example of Set operation

Hive > set env:HOME;env:HOME=/home/hadoophive > set env:HIVE_HOME;env:HIVE_HOME=/usr/local/hive1.2.0hive >

The command space is specified as

Sample code:

# hive-- hivevar column=name#hive-- hiveconf hive.cli.print.current.db=true hive.cli.print.header=true#system: configuration properties defined by java, such as system:user.name (that is, the content of System's properties) # env:shell environment variables, such as env:USER, env.HIVE_HOME## authentication hivevar, system, env [hadoop@nnode hive1.2.0] $hive-- hivevar column=namehive > create table test (id int, ${hivevar:column} string, ${system:user.name} string, path string) OKTime taken: 1.687 secondshive > insert into table test values (1000, 'lisi',' root','${env:HOME}'); # mapreduce process slightly hive > set hive.cli.print.header=true; # display headerhive > select * from test OKtest.id test.name test.hadoop test.path # gets the value of the variable 1000 lisi root / home/hadoop Time taken: 0.147 seconds, Fetched: 1 row (s) hive > # # verify hiveconf [hadoop@nnode hive1.2.0] $hive-- hiveconf hive.cli.print.current.db=truehive (default) > use mywork;OKTime taken: 1.089 secondshive (mywork) > set hive.cli.print.header=true;hive (mywork) > select eno, ename from employee OKeno ename1000 zhangsanTime taken: 0.223 seconds, Fetched: 1 row (s) hive (mywork) >

2. Web mode of Hive

HWI is the abbreviation of Hive Web Interface and is a web replacement scheme of hive cli.

The command to start the hwi service through service is bin/hive-- service hwi

[hadoop@nnode hive1.2.0] $bin/hive-- service hwils: cannot access / usr/local/hive1.2.0/lib/hive-hwi-*.war: No such file or directory15/12/12 22:30:08 INFO hwi.HWIServer: HWI is starting up15/12/12 22:30:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via org.mortbay.log.Slf4jLog15/12/12 22:30:10 INFO mortbay.log: jetty-6 .1.2615 / 12 22:30:11 INFO mortbay.log: Started SocketConnector@0.0.0.0:9999 [hadoop@nnode hive1.2.0] $

It is suggested that the war file of hwi does not exist. The war file of hwi cannot be found when viewed by the find command. It is processed by typing the files in the hwi/web directory under src into war packages:

E:\ Hive\ apache-hive-1.2.0-src\ hwi\ web > jar cvf hive-hwi-1.2.0.war. / *

Upload the war file to the $HIVE_HOME/lib directory, and modify the configuration parameters of hwi through the configuration file (hive-site.xml)

Hive.hwi.listen.host nnode This is the host address the Hive Web Interface will listen on hive.hwi.listen.port 9999 This is the port the Hive Web Interface will listen on hive.hwi.war.file lib/hive-hwi-1.2.0.war This sets the path to the HWI war file, relative to ${HIVE_HOME}.

Start hwi's service service again and access it through web

In fact, tools.jar has been added to the classpath path. We have no choice but to copy the tools.jar under the jdk path to the lib directory of hive, and then start it again through hive-- service hwi. After startup, access, OK.

It mainly includes:

USER user information, including: user authentication (Authorize) creation session (Create Session) session management (List Sessions) DATABASE browsing database and database tables, similar to show databases;show tables;describe table; DIAGNOSTICS to view system diagnostic information, such as the value of System.getProperties, etc.

User authentication in hwi requires entering a user name and user group, such as:

Each user authentication (Authorize) information corresponds to a set of sessions (session). After hive restarts, session information will be lost.

You need to create a session through Create Session before executing the query, and you can view the created session through List Session.

Through List Session, you can see the session group corresponding to the authenticated user (it doesn't really feel useful).

Click Manager to execute the query operation

Submit a query (Submit), which displays the execution status of the session in Session Details, and you can view the results through View File

Conclusion: personal feeling is useless, and it is not as convenient as cli.

3. Remote service of Hive

In fact, you start the hive service as a server, then connect to the hive through JDBC, submit the SQL statements that need to be executed, and return the results after parsing through hive.

[hadoop@nnode hive1.2.0] $bin/hive-- service hiveserverStarting Hive Thrift ServerException in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hive.service.HiveServer at java.net.URLClassLoader$1.run (URLClassLoader.java:366) at java.net.URLClassLoader$1.run (URLClassLoader.java:355) at java.security.AccessController.doPrivileged (Native Method) at java.net.URLClassLoader.findClass (URLClassLoader.java:354) at Java.lang.ClassLoader.loadClass (ClassLoader.java:425) at java.lang.ClassLoader.loadClass (ClassLoader.java:358) at java.lang.Class.forName0 (Native Method) at java.lang.Class.forName (Class.java:274) at org.apache.hadoop.util.RunJar.run (RunJar.java:214) at org.apache.hadoop.util.RunJar.main (RunJar.java:136) [hadoop@nnode hive1.2.0] $

There are two kinds of Hive JDBC driver connections, the early one is HiveServer, and the latest one is HiveServer2. The former has many problems, such as security and concurrency, while the latter solves the problems such as security and concurrency very well.

Introduction: Apache Hive 1.0.0 was officially released on February 8, 2015. This version was supposed to be named Hive 0.14.1, but the team felt it was time to name it in the 1.x.y way. However, there are not many changes in this version, and there are two noteworthy things: define API; for HiveMetaStoreClient, remove HiveServer 1, and fully use HiveServer 2. See: https://cwiki.apache.org/confluence/display/Hive/HiveServer

The service launcher is in ${HIVE_HOME} / bin/hiveserver2 and starts in the following ways:

HIVE_HOME/bin/hiveserver2 or $HIVE_HOME/bin/hive-- service hiveserver2 [hadoop@nnode hive1.2.0] $ll bin/total 32-rwxr-xr-x 1 hadoop hadoop 1031 Apr 30 2015 beelinedrwxr-xr-x 3 hadoop hadoop 4096 Jun 28 13:14 ext-rwxr-xr-x 1 hadoop hadoop 7844 May 8 2015 hive-rwxr-xr-x 1 hadoop hadoop 1900 Apr 30 2015 hive-config.sh-rwxr-xr-x 1 hadoop hadoop 885 Apr 30 2015 hiveserver2-rwxr-xr-x 1 hadoop hadoop 832 Apr 30 2015 metatool-rwxr-xr-x 1 hadoop hadoop 884 Apr 30 2015 schematool [hadoop@nnode hive1.2.0] $

Note:

The default hiveserver port is 10000. You can change the default startup port through hive-- service hiveserver-p 10002, which is also the JDBC connection port.

Hiveserver and hwi service cannot be started at the same time.

If the previous code uses HiveServer (or HiveServer1), when HiveServer2 is used in the hive upgrade, the code needs to be adjusted as follows:

Private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver"; change to private static String driverName = "org.apache.hive.jdbc.HiveDriver"; Connection con = DriverManager.getConnection ("jdbc:hive://ip:10002/default", "username", ""); change to Connection con = DriverManager.getConnection ("jdbc:hive2://ip:10002/default", "username", "")

Start the service

[hadoop@nnode hive1.2.0] $bin/hiveserver2

JavaApi call

Package com.lucl.hive;import java.sql.Connection;import java.sql.DriverManager;import java.sql.ResultSet;import java.sql.Statement;/** @ author lucl * * / public class JDBCHiveDriver {private static String driver = "org.apache.hive.jdbc.HiveDriver"; private static String url = "jdbc:hive2://nnode:10000/mywork"; private static String user = "hadoop" / / users here are users of hadoop cluster private static String pwd = ""; public static void main (String [] args) {String sql = "show tables"; try {Class.forName (driver); Connection con = DriverManager.getConnection (url, user, pwd); Statement st = con.createStatement (); ResultSet rs = st.executeQuery (sql) While (rs.next ()) {System.out.println (rs.getString (1));} con.close ();} catch (Exception e) {e.printStackTrace ();}

Console output result of Eclipse

11348 INFO Utils:310-Supplied authorities: nnode:1000011:25:11351 INFO Utils:397-Resolved authority: nnode:1000011:25:11461 INFO HiveConnection:203-Will try to open client transport with JDBC Uri: jdbc:hive2://nnode:10000/myworktable name is: employeetable name is: employee_02table name is: student

For help on using HiveServer2, see:

Https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-UsingJDBC

Summary:

Personally, it doesn't make much sense to operate the Hive production environment through JDBC. The query of hive is mainly based on MapReduce (although some functions have been optimized so that there is no need for MR processing), while MapReduce belongs to an offline computing model, and its timeliness may be poor. It is impossible for ordinary people to receive it when a call is executed for a few minutes or even longer. The better way is to access RDBMS through JDBC.

The main function of Hive may be that it facilitates the processing of HDFS data. After all, those who are familiar with SQL are not familiar with JAVA or MR programming. They belong to the client tools of HDFS, but their capabilities are limited after all.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.