Shulou(Shulou.com)06/01 Report--

This article mainly explains "what is the function of Hive dynamic partition table". Interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is the function of Hive dynamic partition table"?

1. For partitioned tables with a large number of partitions, you can use dynamic partitioning to have hive automatically create the partitioned tables you need:

The above two parameters represent:

Whether to enable dynamic partition insertion mode. Default is false.

Whether to use strict mode. In strict mode, at least one static column must be specified to prevent users from accidentally overriding all partitions at once.

If you use non-strict mode, you need to specify an additional data column to represent the value of the virtual partition column when inserting, such as:

Set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nonstrict;create table test1 (an int); create table test2 (an int) partitioned by (parid int); insert into table test2 partition (parid) select *, par_id from test1

Otherwise, an error similar to the following will occur:

Error: Error while processing statement: FAILED: SemanticException [Error 10044]: Line 1:18 Cannot insert into target table because column number/types are different 'parid': Table insclause-0 has 2 columns, but query has 1 columns. (state=42000,code=10044)

Finally, execute the command: hdfs dfs-ls / user/hive/warehouse/test2, and you can see that the partition folder is automatically created in hdfs.

Note:

It is found that the insert and alter syntax of hive is not flexible enough, such as not supporting insert into..values.., not supporting alter table test add partition (parid int) and so on. In addition, with regard to the use of Load data syntax, dynamic partitioning is not currently supported. Support for data types is also gradually improving, such as date and timestamp types are supported in recent versions.

For more information on hive partitioning tables, see: hive briefly introduces partitioning tables (partition table), including dynamic partitions (dynamic partition) and static partitions (static partition).

The connection command for hive1 is: su-c hive hdfs

Hive2 added a new client:beeline-u jdbc:hive2://localhost/-n hive

Beeline also provides some built-in commands, using "! help" or "?" You can view it. In addition, beeline is more interactive, such as error prompts and data presentation.

two。 The above briefly describes the use of the hive dynamic partitioning table, and the next small demo is to connect to the hive using Java. Hive is not the same as a relational database. It does not store data and does not provide a method to operate the data. If you want to use jdbc to connect to hive, you must start hive-server. The default connection port is 10000. The code is as follows:

Private static void hive () throws Exception {/ / Class.forName ("org.apache.hadoop.hive.jdbc.HiveDriver"); Class.forName ("org.apache.hive.jdbc.HiveDriver"); Connection con = DriverManager.getConnection ("jdbc:hive2://127.0.0.1:10000/default", "hive", ""); PreparedStatement ps = con.prepareStatement ("insert into table test select? From test2 limit 1 "); ps.setString (1, new Timestamp (new java.util.Date (). GetTime ()). ToString ()); ps.executeUpdate (); ps.setString (1," 2011-11-11 11:11:11 "); ps.executeUpdate (); / / stmt.execute (" set hive.support.concurrency=true ") Statement stmt = con.createStatement (); ResultSet rs = stmt.executeQuery ("select * from test"); int columnSize = rs.getMetaData () .getColumnCount (); while (rs.next ()) {for (int I = 1; I)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.