Installation and use of Sqoop data Analysis engine
Installation and use of Sqoop data Analysis engine
= = > what is Sqoop?
Sqoop is an open source data processing engine, which transfers data between Hadoop (Hive) and traditional relational databases (Oracle, MySQL,Postgres, etc.) through JDBC.
HDFS Hive HBase
< JDBC >Oracle, MySQL
= > installation of Sqoop:
1. Unpack the installation package: tar zxf sqoop-1.4.6.bin__hadoop-0.23.tar.gz-C / app
two。 Configure environment variables:
Vim / .bash_profile
SQOOP_HOME=/app/sqoop-1.4.6.bin__hadoop-0.23
Export SQOOP_HOME
PATH=$SQOOP_HOME/bin:$PATH
=
3. Because you need to transfer data to the database through JDBC, you need to put the JDBC toolkit for the database into the lib directory
Open the oracle installation directory: C:\ oracle\ product\ 10.2.0\ db_1\ jdbc\ lib
Copy the ojdbc14.jar file to the bin directory of sqoop: / app/sqoop-1.4.6.bin__hadoop-0.23/bin
= > Sqoop command details: (note: in oracle, the user name and table name should be all capitalized)
Codegen-- > Mapping relational database tables to a Java file, Java class class, and related jar packages
Sqoop codegen\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-- table EMP
=
Create-hive-table-- > generate Hive tables corresponding to the table structure of a relational database
Sqoop create-hive-table\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-- table EMP
-- hive-table emphive
=
Eval-> quickly use SQL statements to operate on relational databases
This allows you to check whether the SQL statement is correct and display the results on the console when using the import tool for data import
Sqoop eval\
-- connect jdbc:oracal:thin:@192.168.10.210:1521:orcl\
-username SCOTT-password oracle
Query "select * from emp"
=
Export-> Import data from hdfs to relational database
Sqoop export\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-- table STUDENTS # table
-- data tables in export-dir / students # HDFS
=
Import-> Import data from database tables into HDFS
Sqoop export\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-username SCOTT-password oracle
-- table EMP\
-- target-dir / emp
-
-> Import data into HBase (you need to create the table first)
Sqoop import\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-table EMP-columns empno, ename, sal, deptno\
-hbase-table emp-hbase-row-key empno-column-family empinfo
Row-key column family
=
Import-all-tables-> Import data from all tables in the database into HDFS
Sqoop import-all-tables\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-- m 1
=
Job-- > A task used to generate a Sqoop. After it is generated, the task is not executed unless the command is used to execute the task.
=
List-databases-> print out all database names of the relational database
Sqoop list-databases\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl
-username SYSTEM-password oracle
=
List-table-> print out all table names in the relational database
Sqoop list-table\
-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\
-- username SCOTT-- password oracle\
-m 1
=
Merge-- > combine the data from different directories in the HDFS and store them in the specified directory
=
Metastore-> record metadata information of Sqoop job
=
Version-> display Sqoop version information
=
