Shulou(Shulou.com)06/03 Report--

Spark can read data in text, csv and rmdb, and has automatic type detection

Public final static String DATA_SEPARATOR_TAB = "\ t"

Session.read () .format ("csv") .option ("delimiter", Constants.DATA_SEPARATOR_TAB) .option ("inferSchema", "true") .option ("header", "true") .option ("encoding", charset) .csv (path) .toDF (columnNames) .write () .mode (mode) .saveAsTable (tempTable)

The red attribute determines whether spark automatically detects the data type. If automatic detection is not enabled, the default is string.

By default, type detection and correspondence will occur when rdbms is imported into spark, but there is a problem with the time type of processing sqlserver.

As shown in the above figure

Only datetime can be recognized and stored as a date type by spark, and others fall into String type, so the performance will be much worse when performing data_formate because of one more step of conversion.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.