Shulou(Shulou.com)06/01 Report--

Most people do not understand the knowledge points of this article "sqoop how to specify the mode of pg library", so the editor summarizes the following contents, detailed contents, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "sqoop how to specify the mode of pg library" article.

Sqoop is a tool for importing and exporting data between hadoop and relational databases. You can import data from a database (such as mysql,oracle) into hdfs through sqoop, or you can export data from hdfs to a relational database. Sqoop imports and exports through Hadoop's MapReduce, thus providing high parallel performance and good fault tolerance.

Sqoop is suitable for the following people:

System and application developers

System administrator

Database administrator

Data analyst

Data engineer

Description

It is very convenient to use sqoop to export and import data, but there is a problem with postgresql (PG library for short). Pg library is a three-tier database--schema--table. If you want to import into a schema, you need to specify a schema. But how does sqoop specify the mode of the pg library?

Solution.

When you encounter a problem, you have to read the document first. The documentation already indicates how to specify the schema of the pg library. Official document address

The documentation has already said that if you need to add-schema to the specified schema, but note that it must be on the command line! Finally! It will not take effect until it is added.

But this is a command-line solution. What if we are using java? Before it is resolved, my java code is written as follows:

Public static boolean ExportCmdInPg (Configuration conf, String tableName, List columns, String hdfsDir,Map dbMap) {try {LogUtils.logInfoPrint ("start Task", logger); List list = new ArrayList (); list.add ("- connect"); list.add (dbMap.get (Constant.DRIVERURL)); list.add ("- username") List.add (dbMap.get (Constant.USER)); list.add ("- password"); list.add (dbMap.get (Constant.PASSWORD)); list.add ("- table"); list.add (tableName); list.add ("- columns"); list.add (StringUtils.join (columns,',')) List.add ("- fields-terminated-by"); list.add ("\ t"); list.add ("- export-dir"); list.add (hdfsDir); list.add ("- m"); list.add ("1"); ExportTool exporter = new ExportTool (); Sqoop sqoop = new Sqoop (exporter) String [] data = list.toArray (new String [0]); if (0 = = data.length) {LogUtils.logErrorPrint ("the sqoop parameter is empty, please check the ExportCmd method!" , logger); return false;} if (0 = = Sqoop.runSqoop (sqoop, data)) {return true;}} catch (Exception e) {LogUtils.logErrorPrint ("ExportCmd import error into HDFS", logger,e);} return false;}

The result was, of course, unsuccessful. Then I will add-schema parameter to try.

Public static boolean ExportCmdInPg (Configuration conf, String tableName, List columns, String hdfsDir,Map dbMap) {try {LogUtils.logInfoPrint ("start Task", logger); List list = new ArrayList (); list.add ("- connect"); list.add (dbMap.get (Constant.DRIVERURL)); list.add ("- username") List.add (dbMap.get (Constant.USER)); list.add ("- password"); list.add (dbMap.get (Constant.PASSWORD)); list.add ("- table"); list.add (tableName); list.add ("- columns"); list.add (StringUtils.join (columns,',')) List.add ("- fields-terminated-by"); list.add ("\ t"); list.add ("- export-dir"); list.add (hdfsDir); list.add ("- m"); list.add ("1"); list.add ("- schema") List.add ("HERO"); ExportTool exporter = new ExportTool (); Sqoop sqoop = new Sqoop (exporter); String [] data = list.toArray (new String [0]); if (0 = = data.length) {LogUtils.logErrorPrint ("the sqoop parameter is empty, please check the ExportCmd method!" , logger); return false;} if (0 = = Sqoop.runSqoop (sqoop, data)) {return true;}} catch (Exception e) {LogUtils.logErrorPrint ("ExportCmd import error into HDFS", logger,e);} return false;}

The result is also unsuccessful, showing that the error is not recognized-schema. In order to make the schema parameter effective, it cost me a lot of effort. Also checked a lot of information, but the information found is not about the java schema settings. So... The ultimate correct solution is:

Public static boolean ExportCmdInPg (Configuration conf, String tableName, List columns, String hdfsDir,Map dbMap) {try {LogUtils.logInfoPrint ("start sqoop to export oracle data to HDFS directory", logger); List list = new ArrayList (); list.add ("- connect"); list.add (dbMap.get (Constant.DRIVERURL)); list.add ("- username") List.add (dbMap.get (Constant.USER)); list.add ("- password"); list.add (dbMap.get (Constant.PASSWORD)); list.add ("- table"); list.add (tableName); list.add ("- columns"); list.add (StringUtils.join (columns,',')) List.add ("- fields-terminated-by"); list.add ("\ t"); list.add ("- export-dir"); list.add (hdfsDir); list.add ("- m"); list.add ("1") / / Note here is-- it's separate, and here the source code is list.add ("- -"); list.add ("--schema"); list.add ("HERO"); ExportTool exporter = new ExportTool (); Sqoop sqoop = new Sqoop (exporter). String [] data = list.toArray (new String [0]); if (0 = = data.length) {LogUtils.logErrorPrint ("the sqoop parameter is empty, please check the ExportCmd method!" , logger); return false;} if (0 = = Sqoop.runSqoop (sqoop, data)) {return true;}} catch (Exception e) {LogUtils.logErrorPrint ("ExportCmd import error into HDFS", logger,e);} return false;}

Of course, you may also use a string array, which will be written in an array way.

/ / here is just an example String [] string = new String [] {"- -", "--schema", "HERO"} above is about "how sqoop specifies the mode of pg library". I believe you all have some understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about it, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.