Shulou(Shulou.com)06/03 Report--

Spark is a rapidly developing distributed parallel data processing framework in recent years. It is very important to understand and master spark for learning big data. But spark depends on function units, so what is its functional programming process? How do we apply it?

1. Functional programming of Spark

Spark depends on the function unit, function is the basic unit of its programming, only input and output, no state and side effect. Its key concept is to use the function as the input of other functions, but the anonymous function is used in the process of using the function, because this function only meets the current calculation, so it does not need to be solidified for other applications.

Pass a function as an argument

Many RDD operations pass functions as arguments. Here we take a look at the pseudocode of the RDD map operation, applying the function fn to each record of the RDD. But this is not a real code that it executes, just look at the logic it handles through this code.

Example: passing named functions

Anonymous function

Anonymous functions are embedded functions without identifiers, which are most suitable for temporary one-off functions. Supported in many programming languages, such as:

(1) Python:lambda x

(2) Scala:x = >

(3) Java 8RU x->

Example: passing anonymous functions

(1) Python

(2) Scala

Example: Java

(1) Python

(2) Scala

As an important sub-item of big data, Spark must be mastered and studied deeply. However, big data is still in its infancy and has not formed a complete and mature theoretical system, which requires us to dig and learn in many directions and channels. Here recommend "big data cn" Wechat public platform, which introduces a lot of big data's related knowledge, very good!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.