Shulou(Shulou.com)06/01 Report--

Pyspark3 operates csv and postgresql+ to solve "No suitable driver" error report. In view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Environment

Postgresql 9.6

Windows 10

Pyspark 3.0.1

Python 3.8

To operate postgresql, you need to download the corresponding jdbc package: https://jdbc.postgresql.org/download.html

Using from pyspark import SparkContext, SparkConffrom pyspark.sql.session import SparkSession# reads csvsc = SparkContext () sqlContext = SparkSession.builder.appName ('enen'). GetOrCreate () raw_date = sqlContext.read.format (' com.databricks.spark.csv') .options (header='true', inferschema='true' Encoding='utf8') .load (ringing. Accord abc.csv') raw_date.show (10) # read postgresql table structure (JDBC) spark = SparkSession\ .builder\ .appName ('Python Spark Postgresql')\ .config ("spark.jars", ". / postgresql-42.2.18.jar")\ .config (' spark.driver.extraClassPath' '. / postgresql-42.2.18.jar')\ .getOrCreate () df = spark.read\ .format ("jdbc")\ .option ("url", "jdbc:postgresql://localhost:5432/db_name")\ .option ("dbtable", "table_name")\ .option ("user", "postgres")\ .option ("password", "123456789")\ .load () df.printSchema () Note

Be sure to set .config ('spark.driver.extraClassPath','. / postgresql-42.2.18.jar') or the error will be as follows:

Java.sql.SQLException: No suitable driver at java.sql.DriverManager.getDriver (Unknown Source) on pyspark3 operation csv and postgresql+ to solve "No suitable driver" error report what to do to share the answer here, I hope the above content can be of some help to you, if you still have a lot of doubts to be solved, you can follow the industry information channel to learn more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.