Hbase hive integration

Due to the needs of the project, hbase is used to store data. Use hive to query the data. So here we need to integrate hbase and hive. Found the next information on the Internet, in fact, it is also quite simple, mainly in the configuration. Let me make a record of the steps of integration.

1. Environment: hadoop1.0.4,hbase0.92.0,hive0.9.0

two。 When the hadoop hbase hive environment is configured (if not, please refer to other relevant documentation), do the following.

Copy hbase-0.92.0.jar hbase-0.92.0-tests.jar to the hive/lib directory (hive is the installation root of hive). If these two packages already exist under hive, please delete them and copy them later. Modify the hive-site.xml file under hive/conf. Add the following at the bottom: hive.metastore.local true controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM hive.exec.scratchdir / data/hive/scratchdir hive.querylog.location / data/hive/logs hive.aux.jars.path file:///home/mt/hive/lib/hive-hbase-handler-0.9.0.jar,file:///home/mt/hive/lib/hbase-0.92.0.jar, File:///home/mt/hive/lib/zookeeper-3.4.3.jar

Note: my hive-site.xml here is a self-created file, which does not exist by default.

Copy the hbase-0.92.0.jar to all hadoop nodes (including master) under hadoop/lib. Copy the hbase-site.xml file under hbase/conf to all hadoop nodes (including master) under hadoop/conf.

3. Now you can try to start hive.

Mt@ubuntu:~/hive$ bin/hive

And then you can do the test.

Create a database recognized by hbase:

CREATE TABLE hbase_table_1 (key int, value string)

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES ("hbase.columns.mapping" = ": key,cf1:val")

TBLPROPERTIES ("hbase.table.name" = "xyz")

The table name of the hbase.table.name defined in hbase

Column families defined by hbase.columns.mapping in hbase

Import data using sql:

Create a new data table for hive

CREATE TABLE pokes (foo INT, bar STRING)

Insert data in bulk

LOAD DATA LOCAL INPATH'. / examples/files/kv1.txt' OVERWRITE INTO TABLE

Import hbase_table_1 using sql

INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=86

View data

Hive > select * from hbase_table_1

Now you can log in to Hbase to view the data.

Hbase (main): 001describe 0 > xyz'

Hbase (main): 002 scan 0 > xyz'

Hbase (main): 003 put 0 > xyz','100','cf1:val','www.360buy.com'

At this point, you can see the data you just inserted in Hbase in Hive.

Hive > select * from hbase_table_1

Hive accesses an existing hbase

Use CREATE EXTERNAL TABLE

CREATE EXTERNAL TABLE hbase_table_2 (key int, value string)

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf1:val")

TBLPROPERTIES ("hbase.table.name" = "some_existing_table")