Shulou(Shulou.com)06/03 Report--

I. Overview of sqoop 1.1 introduction

​ sqoop is used to correlate derived data between structured data stores such as hadoop (hdfs, hive) and relational databases. Sqoop hatched in March 2012 and is now a top Apache project.

Note that 1.99.7 is not compatible with 1.4.6, and the features are incomplete, and it is not intended for production deployment.

1.2 basic principles

Translate the import or export commands into mapreduce programs to achieve.

The main purpose of the translated mapreduce is to customize inputformat and outputformat.

II. Deploy sqoop

First of all, you have to prepare the hadoop and java environment, which will not be repeated here.

Sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz is used here.

Decompression program:

Tar-zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz-C / opt/modules/

Modify the configuration file:

The decompression path for entering sqoop Enter the conf directory mv sqoop-env-template.sh sqoop-env.sh to modify the content: # export HADOOP_COMMON_HOME=export HADOOP_COMMON_HOME=/opt/modules/hadoop-2.8.4#Set path to where hadoop-*-core.jar is available#export HADOOP_MAPRED_HOME=export HADOOP_MAPRED_HOME=/opt/modules/hadoop-2.8.4#Set the path to where bin/hive is available#export HIVE_HOME=export HIVE_HOME=/opt/modules/hive-1.2.1-bin#Set The path for where zookeper config dir is#export ZOOCFGDIR=export ZOOCFGDIR=/opt/modules/zookeeper-3.4.10/conf

Prepare the connection driver for the relational database to which you need to connect, using mysql here, so download mysql-connector-java-5.1.27-bin.jar, and then

Cp-a mysql-connector-java-5.1.27-bin.jar / opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/lib

Then configure the environment variables:

Vim / etc/profile.d/sqoop.sh#!/bin/bashexport SQOOP_HOME=/opt/modules/sqoop-1.4.6.bin__hadoop-2.0.4-alphaexport PATH=$PATH:$ {SQOOP_HOME} / bin

Verify sqoop:

Sqoop help III. Sqoop simple use case

Before explaining the use of sqoop in detail, let's familiarize ourselves with some simple examples.

3.1 Import data

​ in Sqoop, the concept of "import" refers to the transfer of data from a non-big data cluster (RDBMS) to a big data cluster (HDFS,HIVE,HBASE, etc.), which is called: import, even with the import keyword.

3.1.1 RDBMS to HDFS

First prepare some data in mysql:

$mysql-uroot-p000000mysql > create database company;mysql > create table company.staff (id int (4) primary key not null auto_increment, name varchar, sex varchar); mysql > insert into company.staff (name, sex) values ('Thomas',' Male'); mysql > insert into company.staff (name, sex) values ('Catalina',' FeMale')

Import all data:

Sqoop import\-- connect jdbc:mysql://bigdata121:3306/company\ jdbc connection string-- username root\ user name-- password 000000\ password-- table staff\ table of the data source-- target-dir / user/company\ Hdfs specified directory-- delete-target-dir\ target directory exists, delete-- num-mappers 1\ map number-- fields-terminated-by "\ t" output to hdfs delimiter

Query Import:

Sqoop import\-- connect jdbc:mysql://bigdata121:3306/company\-- username root\-- password 000000\-- target-dir / user/company\-- delete-target-dir\-- num-mappers 1\-- fields-terminated-by "\ t"\ the field separator written to the hdfs file-- query 'select name,sex from staff where id

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.