Shulou(Shulou.com)06/01 Report--

As a bridge between Hadoop and traditional database, Sqoop plays an important role in data import and export. Through the exposition of the basic grammar and functions of Sqoop, we deeply decrypt the function and value of Sqoop.

What is Apache Sqoop?

The Apache open source project developed by Cloudera is an abbreviation for SQL-to-Hadoop. Mainly used in Hadoop (Hive) and traditional databases (mysql, postgresql...) Data transfer between can import the data from a relational database (such as MySQL, Oracle, Postgres, etc.) into the HDFS of Hadoop, and also import the data from HDFS into the relational database. When importing data, you can import not only the entire database, but also a single table, all tables or part of the data into HDFS, and import the data into various formats at the same time. In addition, Sqoop can export data from HDFS to the database, so sqoop has a two-way effect.

2. How does Sqoop work?

Sqoop work is also based on the computing framework MapReduce,MapReduce converts submitted SQL derivatives into MapReduce jobs and then submits them to the cluster. Overall, it is divided into three steps:

1. Check list details

2. Create and submit jobs to the cluster

3. Get the table record and write the data to HDFS

Third, basic grammar:

Sqoop is a command line tool with many subcommands, and there are many tools for importing, exporting, listing database contents, and so on. Usually we list the tables in the database first, and then import the corresponding tables. If you want to see all the tools in sqoop, you can run sqoop help to see a list of all the tools, and it is also recommended that you get used to using help to find some usage parameters. Alternatively, you can run sqoop help tool-name to see help for specific tools (such as import,export).

Basic syntax for Sqoop calls

Fourth, use Sqoop to explore the database:

Usually, before we import the data, we need to explore the database to find out what tables are in the database. There are usually two ways:

1. List-tables: list the tables of all loudacre databases in MySQL

2.eval: execute database query

Sqoop as an important member of the Hadoop family, we need to focus on cognition and grasp, which is of great help for us to understand and learn from big data. Usually, you can also pay attention to some information about big data and understand the development trend of big data and some classic cases in the industry. Especially now the network media is booming and the information spreads rapidly, such as the Wechat service number. I usually like to find some cases from above, such as big data cn is very good. But there is always a threshold for technology, and only by continuous learning and communication can we make continuous progress. Recently, I have learned some experiences shared by others in the service number of big data's Learning Center, which is also of great help to me personally.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.