Shulou(Shulou.com)06/01 Report--

This article shows you how to deploy sqoop. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

1.sqoop download

Https://mirrors.tuna.tsinghua.edu.cn/apache/sqoop/1.4.7/

2.sqoop uploads to the server and decompresses to the appropriate directory

3. Modify the configuration file of sqoop

4. Modify the configuration file

5. Copy the mysql database driver required by sqoop

Cp / home/nflow/servers/hive/lib/mysql-connector-java-5.1.26-bin.jar / home/nflow/servers/sqoop-1.4.7/lib/

6. Start the sqoop test (you can see that you are connecting to the database)

. / sqoop list-databases-- connect jdbc:mysql://127.0.0.1:3306/-- username root-- password 123456

7.sqoop Import data

The sql brought from the blogger

Drop database if exists userdb;create database userdb;use userdb;drop table if exists emp;drop table if exists emp_add;drop table if exists emp_conn; CREATE TABLE emp (id INT NOT NULL,name VARCHAR, deg VARCHAR, salary BIGINT,dept VARCHAR (50)); CREATE TABLE emp_add (id INT NOT NULL,hno VARCHAR (50), street VARCHAR (50), city VARCHAR (50); CREATE TABLE emp_conn (id INT NOT NULL,phno VARCHAR (50), email VARCHAR (50)) Insert into emp values (1201pr); insert into emp values (1202); insert into emp values (1203); insert into emp values (1204); insert into emp values (1205); insert into emp values (1202); insert into emp values (1203); insert into emp values (1204). Insert into emp_add values (1202 288A); insert into emp_add values (1202); insert into emp_add values (1203); insert into emp_add values (1204); insert into emp_add values (1204); insert into emp_add values (1205). Insert into emp_conn values (1202 pr. 2356742); insert into emp_conn values (1202, 1661663); insert into emp_conn values (1203, 8887776); insert into emp_conn values (1204, 9988774); insert into emp_conn values (1205, 1231231) -# Thank you for the copyright of this blogger. I'm just trying out the copyright notice: this is the original article of CSDN blogger "recording every note", which is in accordance with the CC 4.0BY-SA copyright agreement. Please attach the original source link and this notice when reproduced. Original link: https://blog.csdn.net/yumingzhu1/article/details/80678525

From mysql to hdfs

# / bin/bash./bin/sqoop import\-- connect jdbc:mysql://192.168.249.10:3306/userdb\-- username root\-- password 123456\-- table emp\-- M1 [nflow@hadoop-master1 sqoop-1.4.7] $pwd/home/nflow/servers/sqoop-1.4.7 [nflow@hadoop-master1 sqoop-1.4.7] $default export location is / usr/ user / table name Localhost or 127.0.0.1 cannot be used in the database, or an error will be reported and an IP address must be used.

The HDFS directory cannot be repeated in import next time.

Modify the script as follows so that a new one can be generated each time

Import mysql data into hive

Database data

. / sqoop import\-- connect jdbc:mysql://192.168.249.10:3306/userdb\ # userdb--username admin\ # Database admin user-- password 123456\ # password of the database admin user-- table emp_add\ # emp_ addtable in the database admin-- delete-target-dir\ # each deletion- -num-mappers 1\ # # number of mapreduce processes-- hive-import\ # # specifies the default database of hive--hive-database default\ # # hive-- Table name of default data in hive-table empadd\ # # hive-- fields-terminated-by'\ t'# Line feeds

The test result of sqoop importing the same table again is that if the database has new data, then hive will also take it, resulting in repetition as shown in the figure below. How to avoid this problem? incremental synchronization of sqoop

# sqoop incremental synchronization to hive

Id greater than 1207 will synchronize and will not cause duplication.

. / sqoop import\-- connect jdbc:mysql://192.168.249.10:3306/userdb\-- username admin\-- password 123456\-- table emp_add\-- num-mappers 1\-- hive-import\-- hive-database default\-- hive-table empadd\-- fields-terminated-by'\ t'\-- incremental append\-- check-column id\-- last-value 1207

The above is how to deploy sqoop. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.