Spark accumulator experiment

The following code is done with Pyspark + IPython

Count the number of blank lines in the log:

Read the log and create a RDD:

Myrdd = sc.textFile ("access.log")

Do not use accumulators:

In [68]: s = 0In [69]: def f (x):...: global s...: if len (x) = = 0:...: s + = 1.: In [70]: myrdd.foreach (f) In [71]: print (s)

The results are as follows:

0

The reason is that variables of python, even global variables, cannot be applied to synchronize data in various computing processes (threads), so variables of distributed computing framework are needed to synchronize data. Accumulators are used in Spark to solve the problem:

Use accumulator

In [64]: s = sc.accumulator (0) In [65]: def f (x):...: global s...: if len (x) = = 0:...: s + = 1.: In [66]: myrdd.foreach (f) In [67]: print (s)

Get the right results:

fourteen