Shulou(Shulou.com)06/03 Report--

The following mainly brings you a simple way to synchronize data from mysql to hbase using sqoop. I hope that the simple method of synchronizing data from mysql to hbase using sqoop can bring you practical use, which is also the main purpose of my editing this article. All right, don't talk too much nonsense, let's just read the following.

-sqoop- data synchronization-hbase

Use sqoop to synchronize data from mysql to hbase

Knowledge points-the first experience of HBase

Knowledge points-happily accessing HBase through Thrift

Knowledge points-HBaseClient (Java)

The sqoop version currently used by the machine

# sqoop versionWarning: / usr/hdp/2.5.3.0-37/accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.17/12/13 22:34:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.3.0-37Sqoop 1.4.6.2.5.3.0-37git commit id 16c017e90fdde118be73d46eff10cf2c068387e9Compiled by jenkins on Wed Nov 30 02:29:18 UTC 2016

Sqoop1.4.6 document https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html

Sqoop is a tool for data migration between Hadoop and relational databases or mainframes.

You can use Sqoop to import data into a Hadoop distributed file system (HDFS) from relational database management systems such as MySQL and Oracle, or you can export data from HDFS to RDBMS.

RDBMS needs to provide the schema (database connection, username and password and other information) where the data is located, Sqoop uses MapReduce to import and export data, and MR provides fault-tolerant mechanism and parallel operations.

Sqoop is an open source software product of the Apache Foundation, the official website http://sqoop.apache.org

Github address https://github.com/apache/sqoop

Use an example

Sqoop import "- Dorg.apache.sqoop.splitter.allow_text_splitter=true"\-- driver com.mysql.jdbc.Driver\-- connect jdbc:mysql://host:3306/databasename\-- username root\-- password root\-- query "select a.id, a.create_date from table_a a WHERE 1 AND\ $CONDITIONS"-- hbase-table test_sqoop\-- column-family 'fts'\-- hbase-row-key id-- split-by create_date-m 3

7.2.13. Importing Data Into HBase

-- hbase-table specifies the HBase table.

The data entered in each row is converted to the Put operation of HBase as the row output to the HBase table.

By default, Sqoop uses the column [create_date above] of-- split-by as the rowkey.

You can also specify the column as the rowkey through-- hbase-row-key.

Each output row is placed in the same column family, which must be specified through-- column-family.

Cannot use-- direct [conflicts].

-- hbase-row-key can also specify composite row keys, separated by commas.

By default, if the HBase table and column families do not exist, Sqoop will directly report an error to exit, so you need to create the HBase table and column families before Sqoop execution.

If you specify-- hbase-create-table and column families, the HBase table is automatically created if it does not exist.

Sqoop serializes all values and stores an array of bytes converted to utf-8 as a string into HBase.

Sqoop skips all columns that are MULL except the row key.

Specify-- hbase-bulkload can use bulk loading instead of writing directly to HBase

Do you find it very helpful to use sqoop to synchronize data from mysql to hbase? If you need to know more, please continue to follow our industry information. I'm sure you'll like it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.