Shulou(Shulou.com)06/03 Report--

Brief introduction:

Apache Sqoop is an effective tool designed for data conversion tools between Apache Hadoop and structured data stores such as relational databases. You can use Sqoop to import data from external structured data stores into the Hadoop distributed file system or related systems such as Hive and HBase. Instead, Sqoop can be used to extract and export data from Hadoop to external structured data stores such as relational databases and enterprise data warehouses.

Sqoop is designed for big data batch transfer, able to split datasets and create Hadoop tasks to handle each chunk.

Here are the steps to install the deployment:

1. Download the installation package and unzip it

Download address the author is using the sqoop-1.4.6-cdh6.7.0.tar.gz version

# wget download and install package can also be downloaded locally and uploaded to Linux [hadoop@hadoop000 software] $pwd/home/hadoop/software [hadoop@hadoop000 software] $wget http://archive.cloudera.com/cdh6/cdh/5/sqoop-1.4.6-cdh6.7.0.tar.gz# decompression [hadoop@hadoop000 software] $tar-xzvf sqoop-1.4.6-cdh6.7.0.tar.gz-C / home/hadoop/app/# to check See the decompression directory [hadoop@hadoop000 sqoop-1.4.6-cdh6.7.0] $ls-lhtotal 1.9Mdrwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 bin-execute script directory-rw-rw-r-- 1 hadoop hadoop 60K Mar 24 2016 build.xml-rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 cdh.build.properties-rw-rw-r-- 1 hadoop hadoop 35K Mar 24 2016 CHANGELOG.txtdrwxr-xr-x 4 hadoop Hadoop 4.0K Jul 3 16:00 cloudera-rw-rw-r-- 1 hadoop hadoop 6.8K Mar 24 2016 cloudera-pom.xml-rw-rw-r-- 1 hadoop hadoop 9.7K Mar 24 2016 COMPILING.txtdrwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 conf-profile directory drwxr-xr-x 5 hadoop hadoop 4.0K Jul 3 16:00 docs-document drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 ivy-rw-rw -hadoop hadoop 17K Mar 24 2016 ivy.xmldrwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 lib-lib dependency package-rw-rw-r-- 1 hadoop hadoop 15K Mar 24 2016 LICENSE.txt-rw-rw-r-- 1 hadoop hadoop 505 Mar 24 2016 NOTICE.txt-rw-rw-r-- 1 hadoop hadoop 19K Mar 24 2016 pom-old.xml-rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 README.txt-rw- Rw-r-- 1 hadoop hadoop 1012K Mar 24 2016 sqoop-1.4.6-cdh6.7.0.jar-complete jar package-rw-rw-r-- 1 hadoop hadoop 6.5K Mar 24 2016 sqoop-patch-review.py-rw-rw-r-- 1 hadoop hadoop 641K Mar 24 2016 sqoop-test-1.4.6-cdh6.7.0.jardrwxr-xr-x 7 hadoop hadoop 4.0K Mar 24 2016 src-Source drwxr-xr-x 4 hadoop hadoop 4.0K Jul 3 16:00 testdata2. Configure environment variable # add sqoop environment variable to global or only configure personal environment variable [hadoop@hadoop000 ~] $sudo vi/etc/profileexport SQOOP_HOME=/home/hadoop/app/sqoop-1.4.6-cdh6.7.0export PATH=$SQOOP_HOME/bin:$ path [Hadoop @ hadoop000 ~] $source / etc/profile3. Modify the sqoop configuration file [hadoop@hadoop000 conf] $pwd/home/hadoop/app/sqoop-1.4.6-cdh6.7.0/conf [hadoop@hadoop000 conf] $cp sqoop-env-template.sh sqoop-env.sh# to add hadoop and hive directory [hadoop@hadoop000 conf] $vi sqoop-env.sh#Set path to where bin/hadoop is available export HADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0#Set path to where Hadoop-*-core.jar is availableexport HADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0#set the path to where bin/hbase is available#export HBASE_HOME=#Set the path to where bin/hive is availableexport HIVE_HOME=/home/hadoop/app/hive-1.1.0 -cdh6.7.0#Set the path for where zookeper config dir is # export ZOOCFGDIR= 4. Copy the jdbc driver package to the sqoop/lib directory # copy the mysql driver package under the Hive lib directory [hadoop@hadoop000 lib] $pwd/home/hadoop/app/sqoop-1.4.6-cdh6.7.0/lib [hadoop@hadoop000 lib] $cp / home/hadoop/app/hive-1.1.0-cdh6.7.0/lib/mysql-connector-java-5.1.46.jar .5.sqoop simple test using # View command Help [hadoop@hadoop000 ~] $sqoop helpWarning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hbase does not exist! HBase imports will fail.Please set $HBASE_HOME to the root of your HBase installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../zookeeper does not exist! Accumulo imports will fail.Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.18/07/03 16:23:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh6.7.0usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS import-mainframe Import datasets from a mainframe server to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version informationSee 'sqoop help COMMAND' for information on a specific command . # check the sqoop version [hadoop@hadoop000 ~] $sqoop versionWarning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hbase does not exist! HBase imports will fail.Please set $HBASE_HOME to the root of your HBase installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../hcatalog does not exist! HCatalog jobs will fail.Please set $HCAT_HOME to the root of your HCatalog installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.Warning: / home/hadoop/app/sqoop-1.4.6-cdh6.7.0/../zookeeper does not exist! Accumulo imports will fail.Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.18/07/03 16:23:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh6.7.0Sqoop 1.4.6-cdh6.7.0git commit id Compiled by jenkins on Wed Mar 23 11:30:51 PDT 2011The warning here is because I do not have hbase,zookeeper,HCatalog configured

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.