Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to find the oldest suitor in social networks by Spark Graphx". Many people will encounter this dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Spark Graphx provides mapReduceTriplets to aggregate the graph, but it is no longer recommended after 1.2. the source code is as follows:

@ deprecated ("use aggregateMessages", "1.2.0") def mapReduceTriplets [A: ClassTag] (mapFunc: EdgeTriplet [VD, ED] = > Iterator [(VertexId, A)], reduceFunc: (a, A) = > A, activeSetOpt: Option [(VertexRDD [_], EdgeDirection)] = None): VertexRDD [A] * Aggregates values from the neighboring edges and vertices of each vertex. The user supplied* `mapFunc` function is invoked on each edge of the graph, generating 0 or more "messages" to be* "sent" to either vertex in the edge. The `reduceFunc` is then used to combine the output of* the map phase destined to each vertex.** This function is deprecated in 1.2.0 because of SPARK-3936. *

AggregateMessages is recommended:

Def aggregateMessages [A: ClassTag] (sendMsg: EdgeContext [VD, ED, A] = > Unit, mergeMsg: (a, A) = > A, tripletFields: TripletFields = TripletFields.All): VertexRDD [A] = {aggregateMessagesWithActiveSet (sendMsg, mergeMsg, tripletFields, None)}

A simple example is given:

* vertex* {{* val rawGraph: Graph [_, _] = Graph.textFile ("twittergraph") * val inDeg: RDD [(VertexId, Int)] = * rawGraph.aggregateMessages [Int] (ctx = > ctx.sendToDst (1), _ + _) *}

You can see that message passing and aggregation operations can be performed.

Case study: ask for the average age of the oldest suitors and suitors on social networks:

Val oldestFollower: VertexRDD [(String,Int)] = userGraph.aggregateMessages [(String,Int)] (triplet = > {triplet.sendToDst (triplet.srcAttr.name, triplet.srcAttr.age)}, (a) B) = > if (a.room2 > b.room2) an else b) oldestFollower.collect.foreach (println (_)) averageAge: VertexRDD [] = userGraph.aggregateMessages [()] (triplet = > {triplet.sendToDst (triplet.srcAttr.age)} (ab) = > (a.room1 + b.room1) (a.room2 + b.room2)) .mapValues ((idp) = > p.room2 / p.room1) averageAge.collect () .mapValues ((_))

It's good and powerful!

The results are as follows:

Aggregation operation

* * *

Find the oldest suitor:

(4, (Bob,27))

(1, (David,42))

(6, (Charlie,65))

(2, (Charlie,65))

(3, (Ed,55))

* * *

Find out the average age of the suitor:

(4pr 27.0)

(1pr 34.5)

(6pr 60.0)

(2pr 60.0)

(3pr 55.0)

* * *

That's all for "how to find the oldest suitor in social networks with Spark Graphx". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.