Shulou(Shulou.com)05/31 Report--

This article shows you how to carry out scala implicit conversion and Spark source code analysis, the content is concise and easy to understand, absolutely can make your eyes bright, through the detailed introduction of this article, I hope you can get something.

Implicit conversion is an important feature of scala. Today we combine our own examples and spark source code to try to have a more in-depth understanding of implicit conversion.

There are three ways to use implicit, implicit def, implicit class and implicit parameters

First, take a look at the usage of implicit def:

Object implicitTest {

Class Man (val name: String) {

Def work () {println (name + "is working!")}

}

Class Woman (val name: String) {

Def shop () {println (name + "is shopping!")}

}

Class Child (val name: String) {

Def play () {println (name + "is playing!")}

}

Implicit def manToWoman (man: Man) = new Woman (man.name)

Implicit def manToChild (man: Man) = new Child (man.name)

Def main (args: Array [String]): Unit = {

Val man = new Man ("Tom")

Val woman = new Woman ("Lily")

Val child = new Child ("baby")

Man.work ()

Man.shop ()

Man.play ()

}

}

In the above code, we define three class,Man, Woman, Child, and there is no inheritance relationship between them. In the following main function, we declare Man, Woman, Child, an example each, we consider, as a man, is not necessarily only work, appropriate leisure and entertainment is also possible. So we did two implicitly converted functions, manToWoman and manToChild, and it's important to note that these two function names are not important at all, we just want to look easier to understand, you can write an aaa or bbb, there's no problem at all.

The compilation process goes something like this:

1. Compile man.shop () and find that there is no shop method. An error is about to be reported.

2. Before reporting an error, search the scope for implicit conversion functions that can support man's call to shop and find manToWoman.

3. Change man to woman and call the shop method

Then take a look at the usage of implicit class:

Object implicitTest {

Class Man (val name: String) {

Def work () {println (name + "is working!")}

}

Class Woman (val name: String) {

Def shop () {println (name + "is shopping!")}

}

Class Child (val name: String) {

Def play () {println (name + "is playing!")}

}

Implicit class ManConvert (m: Man) {

Def shop () = new Woman (m.name). Shop ()

Def play () = new Child (m.name). Play ()

}

Def main (args: Array [String]): Unit = {

Val man = new Man ("Tom")

Val woman = new Woman ("Lily")

Val child = new Child ("baby")

Man.work ()

Man.shop ()

Man.play ()

}

}

The effect of this seems to be similar to that of the previous one, but in fact its semantics are completely different. The compilation process goes something like this:

1. Compile man.shop () and find that there is no shop method. An error is about to be reported.

2. Before reporting an error, search the scope to see if there are any implicit conversion classes that can provide shop methods for man and find ManConvert

3. Convert man to ManConvert and execute the shop method.

Finally, let's take a look at the use of the implicit parameter:

Object implicitTest {

Class Man (val name: String) {

Def work () {println (name + "is working!")}

Def marry (implicit w: Woman) {println (name + "marryed" + w.name)}

}

Class Woman (val name: String) {

Def shop () {println (name + "is shopping!")}

}

Def main (args: Array [String]): Unit = {

Val man = new Man ("Tom")

Implicit val woman = new Woman ("Lily")

Man.marry

}

}

We have added a marry method to Man, whose parameter w is implicit, and then when we call it, we first declare a woman of type implicit, and then call marry without specifying parameters, so that we will not get an error. But if we declare two woman of implicit in front of marry, then marry will still report an error.

Change to human language to explain this: if a man wants to get married, if he appoints a marriage partner, that's no problem. If not, then see if there is a suitable marriage partner around him. If so, just cook cooked rice directly. But if there are two around him, well, I don't know which one to marry!

Combined with the Spark source code, let's take a closer look at implicit and take a look at the following code:

Val rdd = sc.textFile ("hdfs://master:9000/woozoom/mavlink1.log" 12) .zipWithIndex ()

Rdd.sortByKey ()

It seems to have achieved its goal, but what exactly does this sortByKey mean, in reverse order? In the right order? Can you customize the sorting principles? Look for spark's api documentation, find the RDD class, look for the sortByKey method, but, unexpectedly, there is no, only sortBy, no sortByKey. Confused ~

Until I learned about implicit, and we found the following paragraph in the source code of RDD:

Implicit def rddToOrderedRDDFunctions [K: Ordering: ClassTag, V: ClassTag] (rdd: RDD [(K, V)])

: OrderedRDDFunctions [K, V, (K, V)] = {

New OrderedRDDFunctions [K, V, (K, V)] (rdd)

}

Above we have done a more comprehensive understanding of scala's implicit, such a work, the study and understanding of scala and spark, is very important.

However, I would say that everything has its two sides. While implicit brings compactness and refinement of the code, the negative effect is extremely poor readability; especially in the early stages of work, I strongly do not recommend the extensive use of implicit within our team. In my opinion, writing two more lines of code and explicitly converting it is really no big deal!

The above content is how to carry out scala implicit conversion and Spark source code parsing, have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.