Shulou(Shulou.com)06/03 Report--

Installation and use of Sqoop data Analysis engine

= = > what is Sqoop?

Sqoop is an open source data processing engine, which transfers data between Hadoop (Hive) and traditional relational databases (Oracle, MySQL,Postgres, etc.) through JDBC.

HDFS Hive HBase

Oracle, MySQL

= > installation of Sqoop:

1. Unpack the installation package: tar zxf sqoop-1.4.6.bin__hadoop-0.23.tar.gz-C / app

two。 Configure environment variables:

Vim / .bash_profile

SQOOP_HOME=/app/sqoop-1.4.6.bin__hadoop-0.23

Export SQOOP_HOME

PATH=$SQOOP_HOME/bin:$PATH

=

3. Because you need to transfer data to the database through JDBC, you need to put the JDBC toolkit for the database into the lib directory

Open the oracle installation directory: C:\ oracle\ product\ 10.2.0\ db_1\ jdbc\ lib

Copy the ojdbc14.jar file to the bin directory of sqoop: / app/sqoop-1.4.6.bin__hadoop-0.23/bin

= > Sqoop command details: (note: in oracle, the user name and table name should be all capitalized)

Codegen-- > Mapping relational database tables to a Java file, Java class class, and related jar packages

Sqoop codegen\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-- table EMP

=

Create-hive-table-- > generate Hive tables corresponding to the table structure of a relational database

Sqoop create-hive-table\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-- table EMP

-- hive-table emphive

=

Eval-> quickly use SQL statements to operate on relational databases

This allows you to check whether the SQL statement is correct and display the results on the console when using the import tool for data import

Sqoop eval\

-- connect jdbc:oracal:thin:@192.168.10.210:1521:orcl\

-username SCOTT-password oracle

Query "select * from emp"

=

Export-> Import data from hdfs to relational database

Sqoop export\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-- table STUDENTS # table

-- data tables in export-dir / students # HDFS

=

Import-> Import data from database tables into HDFS

Sqoop export\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-username SCOTT-password oracle

-- table EMP\

-- target-dir / emp

-

-> Import data into HBase (you need to create the table first)

Sqoop import\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-table EMP-columns empno, ename, sal, deptno\

-hbase-table emp-hbase-row-key empno-column-family empinfo

Row-key column family

=

Import-all-tables-> Import data from all tables in the database into HDFS

Sqoop import-all-tables\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-- m 1

=

Job-- > A task used to generate a Sqoop. After it is generated, the task is not executed unless the command is used to execute the task.

=

List-databases-> print out all database names of the relational database

Sqoop list-databases\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl

-username SYSTEM-password oracle

=

List-table-> print out all table names in the relational database

Sqoop list-table\

-- connect jdbc:oracle:thin:@192.168.10.210:1521:orcl\

-- username SCOTT-- password oracle\

-m 1

=

Merge-- > combine the data from different directories in the HDFS and store them in the specified directory

=

Metastore-> record metadata information of Sqoop job

=

Version-> display Sqoop version information

=

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.