Shulou(Shulou.com)06/01 Report--

This article mainly explains the "Sqoop2 from Mysql into Hdfs process", the article explains the content is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in-depth, together to study and learn "Sqoop2 from Mysql into Hdfs process"!

I. Environmental construction

1.Hadoop

Http://my.oschina.net/u/204498/blog/519789

2.Sqoop2.x

Http://my.oschina.net/u/204498/blog/518941

3. Mysql

Import hdfs from mysql

1. Create mysql databases, tables, and test data

Xxxxxxxx$ mysql-uroot-pEnter password: mysql > show databases;+-+ | Database | +-+ | information_schema | | mysql | | performance_schema | | test | +-+ 4 rows in set (0.00 sec) test = > New database mysql > use test Mysql > show tables;+--+ | Tables_in_test | +-+ | | test | +-+ 1 rows in set (0.00 sec) test = > is the new table mysql > desc test +-+ | Field | Type | Null | Key | Default | Extra | + -+-+ | id | int (11) | NO | PRI | NULL | auto_increment | | name | varchar (45) | YES | | NULL | | age | int (11) | YES | | NULL | | + -+ 3 rows in set (0.00 sec) mysql > select * from test +-7 | a | 1 | 8 | b | 2 | 9 | c | 3 rows in set (0.00 sec)

two。 Authorize individual users

Note: after sqoop submits the job, each node will access the database during the map phase, so prior authorization is required.

Mysql > grant [all | select |...] On {db}. {table} to {user} @ {host} identified by {passwd}; mysql > flush privileges;# I authorize username:root passwd:root to access any table in db:test to a specific hostname with the permission allmysql > grant all on test.* to 'root'@ {host} identified by' root'

3. Start sqoop2-server

[hadoop@hftclclw0001 sqoop-1.99.6-bin-hadoop200] $pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200 [hadoop@hftclclw0001 sqoop-1.99.6-bin-hadoop200] $. / bin/sqoop2-server start.webui can access the check or view the log

4. Start sqoop2-shell

[hadoop@hftclclw0001 sqoop-1.99.6-bin-hadoop200] $pwd/home/hadoop/sqoop-1.99.6-bin-hadoop200 [hadoop@hftclclw0001 sqoop-1.99.6-bin-hadoop200] $. / bin/sqoop2-shell .sqoop: 000 > show version.sqoop:000 > show connector+----+----+-+ -+-+ | Id | Name | Version | Class | Supported Directions | + -+-- +-+ | 1 | Generic-jdbc-connector | 1.99.6 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector | FROM/TO | | 2 | kite-connector | 1.99.6 | org.apache.sqoop.connector.kite.KiteConnector | FROM/TO | | 3 | hdfs-connector | 1.99.6 | org.apache.sqoop.connector.hdfs.HdfsConnector | FROM/TO | | 4 | kafka- Connector | 1.99.6 | org.apache.sqoop.connector.kafka.KafkaConnector | TO | + -+-- + create connectorsqoop:000 based on your connector > create link-c 1 = > create jdbc first and enter name, Jdbc-driver, url, username, passwd, etc. Sqoop:000 > create link-c 3 = > when creating hdfs, enter name, hdfs url, Wait, sqoop:000 > show link+----+----+-+ | Id | Name | Connector Id | Connector Name | Enabled | +-+-- -+-+ | 3 | 10-21_jdbc1 | 1 | generic-jdbc-connector | true | | 4 | 10-21_hdfs1 | 3 | hdfs-connector | true | +-+-- -+ create job-f = > from-t to, that is, from where to import sqoop:000 > create job-f 3-t 4 will be filled in The corresponding table information. There is also hdfs information sqoop:000 > show job +-+ | Id | Name | From Connector | To Connector | Enabled | +-+ -+ | 1 | 10-20_sqoopy2 | 1 | 3 | true | + -+ # start jobsqoop:000 > start job-j 2. You can access it on webui to check the progress, or you can use sqoop:000 > status job-j 2

Guide of sqoop

Http://sqoop.apache.org/

5.troubleshooting

Read more logs and check them slowly.

Thank you for your reading, the above is the content of "the process of introducing Sqoop2 from Mysql to Hdfs". After the study of this article, I believe you have a deeper understanding of the process of introducing Sqoop2 from Mysql to Hdfs, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.