Shulou(Shulou.com)05/31 Report--

How to use the accumulator accumulator in spark? I believe many inexperienced people don't know what to do about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

When the cumulative result type is the same as the merged element type, it is simpler to accumulate values, that is, variables that are "added" only through association operations, so they can be effectively supported in parallel. They can be used to implement counters (such as MapReduce) or summation. Spark natively supports accumulators of numeric types, and programmers can add support for new types.

Create an accumulator from the initial value v by calling SparkContext#acculator. You can then use the Accumulable#+= operator to add tasks running on the cluster to the cluster. However, they cannot read their values. Only the driver can read the value of the accumulator using its value method.

The following interpreter session shows the accumulator used to add array elements:

Scala > val accum = sc.accumulator (0) accum: spark.Accumulator [Int] = 0scala > sc.parallelize (Array (1,2,3,4)) .foreach (x = > accum + = x)... 10-09-29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 sscala > accum.valueres2: Int = 10 after reading the above, have you mastered how to use accumulator, the accumulator in spark? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.