Shulou(Shulou.com)06/01 Report--

Editor to share with you what to do about the exception from Sqoop to Hive table. I believe most people don't know much about it, so share this article for your reference. I hope you can learn a lot after reading this article. Let's learn about it together.

1. Problem description

When you use Sqoop to extract MySQL data to the Hive table, the extraction statement executes normally but the data is not written to the Hive table. The Sqoop decimation script is executed as follows:

Export HADOOP_USER_NAME=hive

Sqoop import\

-connect "jdbc:mysql://cdh02.fayson.com:3306/cm"\

-- username cm\

-- password password\

-- table HOSTS\

-fields-terminated-by','\

-- target-dir / user/hive/warehouse/hive_hosts\

-- delete-target-dir\

-- hive-import\

-- hive-overwrite\

-- hive-table hive_hosts\

-- m 1\

-- input-null-string'\ N'\

-- input-null-non-string'\ N'

(can slide left and right)

Execute the above Sqoop script on the command line to show that the job is executing normally

The execution result shows that the number of files and the data size are both 0. Log in to the hive_ hosts table generated by the Hive query

two。 Exception handling and analysis

1. There is a target-dir parameter in the script of Sqoop decimation, which points to the HDFS data directory and is the warehouse directory of the hive_hosts table. Remove the target-dir parameter and execute it.

Script executed successfully

As you can see in the log of the script above, write a data file to HDFS with the file size 481Byte.

two。 Use the hive command to view the hive_hosts table, and the data has been extracted into the hive_hosts table normally.

Description and influence of target-dir parameters:

In the Sqoop decimation script, the data extracted by target-dir directory Sqoop is written to the directory specified by target-dir. After Sqoop completes the data extraction, the data from target-dir directory is load to the warehouse directory corresponding to Hive table and the target-dir directory and data are deleted. Because the target-dir directory we specified in the above exception is the same as the warehouse directory of the Hive table, it will cause the Sqoop decimation to succeed, but there is no data problem in the corresponding Hive table.

If the target-dir directory is not specified in the Sqoop script, the extracted data is written to the / user/ {username} / {db_tablename} directory by default.

The above is all the contents of the article "what to do if there is an exception from Sqoop to the Hive table". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.