Shulou(Shulou.com)06/02 Report--

This article mainly explains "the configuration of hive and the description of Metabase table". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn the configuration of hive and the description of Metabase table.

I. hive configuration

The configuration for hive is in the hive-site.xml file.

Property description default value hive.metastore.warehouse.dir specifies the storage path of the hive table on hdfs / connection URLjavax.jdo.option.ConnectionUserNameUserNameConnectionUserNameUserNameUserNameUserusername javax.jdo.option.ConnectionPassword Metabase connection password

For example, the following configuration:

Javax.jdo.option.ConnectionURL jdbc:mysql://m000:3306/hive JDBC connect string for a JDBC metastore javax.jdo.option.ConnectionUserName hiveuser username to use against metastore database javax.jdo.option.ConnectionPassword hiveuser password to use against metastore database

In hive, the newly created database and tables are located under the hive.metastore.warehouse.dir path of HDFS by default. The following figure shows the file paths of database, table, and table files on HDFS, respectively.

According to the database set in javax.jdo.option.ConnectionURL, as well as the user name and password, hive can write and read its metadata information.

There is an indirect way to connect to the Metabase in addition to configuring URL,username,password. You can start a metastore service with hive--service metastore on hive client A, and then configure hive.metastore.uris=thrift://A:9083 in the hive-site.xml file of another hive client B. you can also access metadata information (port 9083 is the default and can be manually specified using the-p parameter).

2. Description of Metabase tables

This section describes the role of some of the more important tables in the hive Metabase, which is gradually added with the subsequent use of Hive.

Tables in the mysql Metabase hive:

The table name functions as BUCKETING_COLS to store bucket field information, to associate a field CD_ID with other tables through SD_ID, to associate COLUMNS_V2 storage field information with SDS tables, to associate DATABASE_PARAMS empty DBS storage field information with other tables through CD_ID, to store database information of empty hive DELETEME1410257703262 empty FUNC_ empty global _ PRIVS variables, table-independent IDXS empty INDEX_PARAMS empty PARTITIONS partition records, SD_ID, TBL_ID associated PARTITION_KEYS storage partition fields TBL_ID associates the value of the PARTITION_KEY_VALS partition, which is associated by PART_ID. Share the same field INTEGER_IDX as PARTITION_KEYS to mark different partition fields. PARTITION_PARAMS stores information about a partition, including the number of files, file size, number of records, and so on. Associate PART_COL_ PRIVS empty part _ COL_ stats empty part _ PRIVS empty ROLES role table through PART_ID, cooperate with GLOBAL_PRIVS, and SDS independent of table stores input and output format and other information, including table format and partition format. The associated field CD_ID,SERDE_IDSD_ params empty sequence _ TABLE stores sqeuence-related information, table-independent SERDES storage serialization deserialization uses class SERDE_PARAMS serialization and deserialization related information, associates SKEWED_COL_ names empty skid _ COL_VALUE_LOC_ map empty ski _ STRING_ LISTempty SKEWED _ STRING_LIST_ values empty SKEWED _ values empty so _ COLS sort field through SERDE_ID, associates TABLE_ params table related information through SD_ID, whether it is an external table or not Associate the TAB_COL_STATS empty TBLS storage table information through TBL_ID, assign permission related information to the associated field DB_ID,SD_ID,TBL_COL_ PRIVS empty TBL _ PRIVS table, associate the VERSION version VERSION_copy version through TBL_ID, and associate it through VER_ID.

Here is an additional introduction to metatool, a tool script for hive. If you need to modify a lot of related records in the Metabase, you can take a specific look at how the metatool script is used.

For example, when you HA a HDFS, if the full path of hdfs is hdfs://m000, and after you set dfs.nameservices to my-cluster after HA, the access path of hdfs becomes hdfs://my-cluster. You need to update all records in the hive Metabase. You can refer to the following actions

Use the metatool script, first the new path, then the old path

/ usr/local/bigdata/hive/bin/metatool-updateLocation hdfs://my-cluster hdfs://m00

At this point, I believe you have a deeper understanding of "hive configuration and Metabase table description". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.