Shulou(Shulou.com)06/03 Report--

The following code is done with Pyspark + IPython

Count the number of blank lines in the log:

Read the log and create a RDD:

Myrdd = sc.textFile ("access.log")

Do not use accumulators:

In [68]: s = 0In [69]: def f (x):...: global s...: if len (x) = = 0:...: s + = 1.: In [70]: myrdd.foreach (f) In [71]: print (s)

The results are as follows:

0

The reason is that variables of python, even global variables, cannot be applied to synchronize data in various computing processes (threads), so variables of distributed computing framework are needed to synchronize data. Accumulators are used in Spark to solve the problem:

Use accumulator

In [64]: s = sc.accumulator (0) In [65]: def f (x):...: global s...: if len (x) = = 0:...: s + = 1.: In [66]: myrdd.foreach (f) In [67]: print (s)

Get the right results:

fourteen

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.