Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to write the code for broadcast variables and accumulators in Spark03. Many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

Broadcast variable scala code

Def main (args: Array [String]): Unit = {val conf = new SparkConf () conf.setMaster ("local") conf.setAppName ("Spark07") val sc = new SparkContext (conf) / / the accumulator defines and assigns the initial value on the driver side. The accumulator can only be read on the driver side and updated on the driver and Excutor side. Var sum= sc.accumulator (0) val rdd: RDD [String] = sc.textFile ("G:/idea/scala/spark02/words1") sum.add (1) val result: RDD [String] = rdd.map (index= > {println ("sum=" + sum) sum.add (1) println (sum) index}) result.foreach (println) println (sum)

Accumulator scala code

Def main (args: Array [String]): Unit = {val conf = new SparkConf () conf.setMaster ("local") conf.setAppName ("Spark07") val sc = new SparkContext (conf) / / the accumulator defines and assigns the initial value on the driver side. The accumulator can only be read on the driver side and updated on the driver and Excutor side. Var sum= sc.accumulator (0) val rdd: RDD [String] = sc.textFile ("G:/idea/scala/spark02/words1") sum.add (1) val result: RDD [String] = rdd.map (index= > {println ("sum=" + sum) sum.add (1) println (sum) index}) result.foreach (println) println (sum)} read the above Do you have any further understanding of how to write the code for broadcasting variables and accumulators in Spark03? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.