How to analyze the variable problem in SPARK foreach Loop
How to resolve the variable problem in SPARK foreach loop, for this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more small partners who want to solve this problem find a simpler and easier way.
reason
The external variable referenced in the spark operator is actually a copy of the variable, and modifying its value in the operator only changes the value of the copy, and the external variable remains unchanged.
In plain English, variables in foreach cannot be brought out unless map is used to return the result as rdd.
Solution: 1. Use the broadcast variable object foreachtest { def main(args: Array[String]): Unit = { val conf = new SparkConf() conf.setMaster("local[1]") conf.setAppName("WcAppTask") val sc = new SparkContext(conf) sc.setLogLevel("WARN") val fileRdd = sc.parallelize(Array(("imsi1","2018-07-29 11:22:23","zd-A"),("imsi2","2018-07-29 11:22:24","zd-A"),("imsi3","2018-07-29 11:22:25","zd-A"))) val result = mutable.Map.empty[String,String] val resultBroadCast: Broadcast[mutable.Map[String, String]] =sc.broadcast(result) fileRdd.foreach(input=>{ val str = (input._ 1+"/t"+input._ 2+"/t"+input._ 3).toString resultBroadCast.value += (input._ 1.toString -> str) println(resultBroadCast.value.size) //Returns 1,2.3 }) println(result.size) //returns 3}2: uses accumulator val accumulator = sc.collectionAccumulator[mutable.Map[String, String]]("My Accumulator")fileRdd.foreach(input => { val str = input._ 1 + "/t" + input._ 2 + "/t" + input._ 3 accum.add(mutable.Map(input._ 1 -> str)})println(accum.value.size())3: cumulative variable longAccumulator val longaa= sc.longAccumulator("count")fileRdd.foreach(input=>{ val str = (input._ 1+"/t"+input._ 2+"/t"+input._ 3). toString longaa.add(1L)})println(longaa.count) //Return 3 The answer to the variable problem in the SPARK foreach loop is shared here. I hope the above content can be helpful to everyone. If you still have a lot of doubts, you can pay attention to the industry information channel to learn more.
