Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how spark connects to relational databases through jdbc". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1 create sparkSession object val sparkSession = SparkSession.builder () .appName ("findIPRel") .getOrCreate () 2 jdbc link create DataFrame object val jdbcDF = sparkSession.read .format ("jdbc") .option ("driver", "org.postgresql.Driver") / / .option ("url", "jdbc:postgresql://121.36.61.51:5432/xsbigdatadb") .option ("dbtable") must be added "ods_proto_log") .option ("user", "postgres") .option ("password", "root@456") .load ()

It should be noted that the driver option must be specified, otherwise even if your classpath contains the jdbc package of the corresponding database, you will not be able to find the appropriate driver error.

3 query the data by dsl through dataFrame object val ori = jdbcDF.select ("srcip", "dstip"). Rdd.map {case Row (col1:String,col2:String) = > (col1,col2) case _ = > (srcIp,srcIp)} 4. Other

Dependency of jdbc newspaper added to pg database in maven

Org.postgresql postgresql 42.2.18

Spark submission specification relies on the jar package to add the-- jars option to the submit command, such as spark-submit-- class com.pxu.spark.core.FindIpRel-- jars. / postgresql-42.2.18.jar-- master spark://master:7077. / sparkLearn-1.0-SNAPSHOT-jar-with-dependencies.jar 192.168.14.17 192.168.73.58 4 hdfs://master:9000/submitTest/output/out3

This is the end of the content of "how spark connects to relational databases through jdbc". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.