Shulou(Shulou.com)05/31 Report--

Today, I will talk to you about how to understand Transformation and Action in Spark. Many people may not know much about it. In order to make you understand better, the editor has summarized the following for you. I hope you can get something according to this article.

1. Two types of RDD operations

Creation of RDD:

(SparkContext) sc.parallelize (collection) / / convert a collection to RDD

Sc.textFile ("path..") / / read a file and convert it to RDD

2. Transformations type

Delayed execution, a RDD through the operation generated by the new RDD will not be executed immediately, only wait until the Action operation will actually be executed.

Map (func): use func for each element in the RDD dataset that calls map, and then return a new RDD. The returned dataset is an example of a distributed dataset: var list = List. / / define a List collection / / for the list that calls map, num represents every element in the list, and each element * 2 element can be understood as a variable name list.map (num= > num*2) result: List (2 num= > num*2) / / list.map (num= > num*2) can be regarded as every element*2filter (func) in the set: func is used for every element in the RDD dataset that calls filter Then an example of RDD consisting of elements that make func true is returned: var list = List (1, 2, 3, 4, 5, 6) / / define a List collection / / for list that calls filter, num represents every element in list, and filter every elementlist.filter (num= > num > 2) / / parameter that meets > 2 criteria must be a Boolean print result: List [Int] = List (3,4,5,6) flatMap (func): similar to map But flatMap generates multiple examples of results: / / define a collection with elements of String var strList = List ("hello world", "hello xm", "hello xh", "hello, Im fine", "hello how are you?") / / split each element into multiple fields, and flatMap each element participating in the calculation must return multiple results or an array For example, the returned result can be as follows: List (Array (...), Array (...),...) strList.flatMap (stsr= > str.split (")) print result: List (hello, world, hello, xm, hello, xh, hello, Im, fine, hello, how, are, you?) groupByKey (): group value of the same key groupByReduce (): merge value of the same key union: aggregate two RDD For example: RDD1- > (1pens 2jue 3), RDD2- > (1pens 2pens 3), RDD1 union RDD2- > (1pje 2pje 3pas 1pas 3) join: take the Cartesian product SortByKey (boolean) of two RDD: sort by key, true in descending order, false in ascending order, if you want to sort value, you can swap the positions of key and value, and exchange operations: map (x = > (x.Zhen2Jing x.coach 1) 3, Actions type

When the Spark job is submitted, when Action, the operation of the Transformation type will actually perform the calculation operation, and then produce the final result output.

Cache: put the results in memory count: count the number of element collect: return an array of results reduce: merge all element lookup: view valuesave for a certain key: save the results to disk and read the above, do you have any further understanding of how to understand Transformation and Action in Spark? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.