Shulou(Shulou.com)06/01 Report--

Sqoop, as a data transfer tool, plays a role as a bridge for data transfer between Hadoop and traditional databases, so how to import and export data?

First: Use MapReduce jobs to perform imports:

(1) Sqoop first checks the table to be imported

1. Determine the primary key (if any), call mareduce, and divide the map according to the primary key.

2. There is no primary key, run boundary query to determine the number of imported records (find a min and a max, determine the boundary to divide)

3. Divide the results of boundary queries by the number of tasks, so that they have the same load

(2) Sqoop generates Java source files for each table to be imported

Compile and use files during import

2. It remains after import and can be safely deleted.

Second: import the entire database using sqoop:

The import-all-tables tool imports the entire database

1. Files stored as comma intervals

2. Import to HDFS home directory by default

Data will be placed in a subdirectory of each table.

(2) Specify a different base directory using the--warehouse-dir option

Third: Import a single table using Sqoop:

(1) The import tool imports a single table

1. Example: Import accounts table

Comma separated data storage to HDFS

2. Specify tab separation fields

Fourth: incremental import

(1) What happens if there are changes since the last record import?

1. You can import all records again, but the efficiency is low.

(2) lastmodified incremental mode of Sqoop imports new and modified records

Based on the specified timestamp column

Ensure that timestamp updates when records are updated or added

(3) append incremental mode only imports new records

1. Based on the last record of the specified column

Fifth: Export Data from Hadoop to RDBMS Using Sqoop

(1) Sqoop's import tool pulls data from RDBMS into HDFS

(2) Sometimes it is also necessary to push HDFS data back to the RDBMS, when it is necessary to batch large data sets and export the results to the RDBMS for access by other systems.

(3) Sqoop With the export tool, RDBMS tables must already exist before export

Mastering the above methods, there is basically a clear understanding and understanding of the import and export of relational databases. In fact, we should look at other people's sharing in the daily learning and practice process. After all, everyone's experience and experience in technology are different, and sometimes it is impossible to intentionally harvest. I usually like to pay attention to "big data cn" and "big data era learning center" these WeChat service numbers, for me personally, played a great role in promoting, but also expect everyone learning big data can gain something!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.