Shulou(Shulou.com)06/03 Report--

There are big differences between sqoop2 and sqoop1. The whole ETL process is divided into two parts: server and client. Sqoop2 is an interactive operation.

Decompression and installation

[hadoop@fa01] $tar-zxvf sqoop-1.99.7-bin-hadoop200.tar.gz-C. /

Rename

[hadoop@fa01 ~] $mv sqoop-1.99.7-bin-hadoop200 sqoop-1.99.7

At the same time, create a new extra file and place the driver file of mysql (the file name is arbitrary, and the configuration file corresponds to the configuration file)

[hadoop@fa01 sqoop-1.99.7] $mkdir extra

[hadoop@fa01 ~] $mv mysql-connector-java-5.1.5-bin.jar / home/hadoop/sqoop-1.99.7/extra/

Configure environment variables. Configure sqoop third-party jar package reference variables

[hadoop@fa01 ~] $vim .bash _ profile

Export SQOOP_HOME=/home/hadoop/sqoopsqoop-1.99.7

Export SQOOP_SERVER_EXTRA_LIB=/home/hadoop/sqoop-1.99.7/extra

Modify the core-site.xml file of hadoop, where hadoop is the user name

[hadoop@fa01 ~] $cd hadoop-2.6.0-cdh6.5.2/etc/hadoop

[hadoop@fa01 hadoop] $vim core-site.xml

Hadoop.proxyuser.hadoop.hosts

*

Hadoop.proxyuser.hadoop.groups

*

Configure # Hadoop configuration directory in the conf file in sqoop

[hadoop@fa01 ~] $cd sqoop-1.99.7/conf/

[hadoop@fa01 conf] $vim sqoop.properties

Org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-2.6.0-cdh6.5.2/etc/hadoop

Check whether sqoop is configured successfully

[hadoop@fa01 bin] $. / sqoop2-tool verify

Setting conf dir: / home/hadoop/sqoop-1.99.7/bin/../conf

Sqoop home directory: / home/hadoop/sqoop-1.99.7

Sqoop tool executor:

Version: 1.99.7

Revision:435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb

Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine

Running tool: class org.apache.sqoop.tools.tool.VerifyTool

2017-06-21 01 INFO 31 core.SqoopServer 48489 [main] core.SqoopServer (SqoopServer.java:initialize (55))-Initializing Sqoop server.

2017-06-21 01 core.PropertiesConfigurationProvider 31 core.PropertiesConfigurationProvider 48539 INFO [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize (99))-Starting config file poller thread

Verification was successful.

Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

Start sqoop

[hadoop@fa01 bin] $. / sqoop2-server start

Setting conf dir: / home/hadoop/sqoop-1.99.7/bin/../conf

Sqoop home directory: / home/hadoop/sqoop-1.99.7

Starting the Sqoop2 server...

2017-06-21 01 INFO [main] core.SqoopServer (SqoopServer.java:initialize (55))-Initializing Sqoop server.

2017-06-21 01 core.PropertiesConfigurationProvider 32 INFO [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize (99))-Starting config file poller thread

Sqoop2 server started.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.