Shulou(Shulou.com)06/01 Report--

In this issue, the editor will bring you about what the formula of spark generation table is. The article is rich in content and analyzed and described from a professional point of view. I hope you can get something after reading this article.

IndexABCD1102882938

After simplifying the business description, the requirements are:

If there is a table as above, column An and the first row can be given directly, and the remaining blank fields are required to be calculated according to the formula.

Formula:

1) B [x] = A [x]-1

2) C [x] = A [x]-B [x]

3) D [x] = (C [x] + D [x-1]) / 2 / (previous row same column + previous column of this row) / 2

The calculated table value should be:

IndexABCD1102882972538444.5

Code

Import org.apache.spark.rdd.RDDimport org.apache.spark.sql.types._import org.apache.spark.sql. {Row, SparkSession} val schema = StructType (List ("ID", DoubleType, true), StructField ("A", DoubleType, true), StructField ("B", DoubleType, true), StructField ("C", DoubleType, true), StructField ("D", DoubleType) (true)) val data1: RDD [Row] = spark.sparkContext.parallelize (Seq (Row (1.0 data1, schema) df1.createOrReplaceTempView ("df1") val data2: RDD [Row] = spark.sparkContext.parallelize (Seq (Row)) val df2 = spark.createDataFrame (data2 Schema) df2.createOrReplaceTempView ("df2") def func_1 (x:Int) = {x-1} def func_2 (x-func_1 (y)} def func_3): Int = {(x+func_2 (y)) / 2} spark.udf.register ("func_1", func_1 _) spark.udf.register ("func_2") Func_2 _) spark.udf.register ("func_3", func_3 _) spark.sql ("| select | df2.ID, | df2.A, | func_1 (df1.D) as B, | func_2 (df2.A,df1.D) as C, | func_3 (df1.D,df2.A) Df1.D) as D | from df2 inner join df1 on df1.ID = df2.ID | "" .stripMargin) .stripMargin). Show () the above is what is the formula for generating the spark table shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.