Shulou(Shulou.com)06/01 Report--

This article is about the use of Spark2.3.1 skills to share with you, the editor feels very practical, so share with you to learn, I hope you can learn something after reading this article, do not say much, follow the editor to have a look.

Spark 2.3.1 use the technique Spark-SQL to read the JSON file when reflecting the header case class StudentInfo (id:Long,name:String,age:Int) val example = spark.read.json ("/ data/result.json") .as (StudentInfo) example.show () dynamically define schema

Different schema needs to be defined according to different data

Val schemaInfo = "name age" val fields = schemaInfo.map (item= > item.split (") .map (item= > StructField (item,StringType,nullable=true)) val schema = StructType (fields) val rowRDD = peopleRDD.map (_ .split (") .map (attributes= > Row (attributes (0), attributes (1)) val peopleDF = spark.createDataFrame (rowRDD,schema) peopleDF.show () Spark 2.3.1 on YARNspark-submit limit parameter is not valid

Because the executor-memory 2g configured during spark-submit did not take effect, he later asked his colleague that he also encountered the same problem, and the solution was to allocate executor dynamically.

-- conf spark.yarn.maxAppAttempts=1-conf spark.yarn.maxAppAttempts=1-- conf spark.dynamicAllocation.minExecutors=2-- conf spark.dynamicAllocation.maxExecutors=4-- the above is what the skills of using Spark2.3.1 are, and the editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.