Shulou(Shulou.com)06/01 Report--

How to combine demo and spark in Scala language to realize chain computing? I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

First, what is chain calculation?

1. Generally speaking, developers are used to encapsulating things into a method; the idea of chain programming is to encapsulate the things to be done in block and provide an external method to return this block.

2. The characteristics of chain programming ideas and methods: the return value of the method must be the parameter of block,block is the content of the operation, and the return value of block is the caller of the method that returns this block

Second, give examples to illustrate

For example, let's define a case class Person

Case class Person (privateval parent: Person = null, privateval name: String = null, private var age: Int = 0) {

Def setName (newName: String) = new Person (this,newName, this.age)

Def setAge (newAge: Int): this.type = {

This.age = newAge

This

}

Def introduce {parentIntroduce; println (s "Hello, my name is $name and I am $age years old.")}

Def parentIntroduce {if (parentless null) parent.introduce}

}

Then, we can do the following:

Person (null, "kitty", 45) .setName ("Peter"). SetAge (41). SetName ("Peter1"). SetAge (21). Introduce

Results of execution

Hello, my name is kitty and I am 45 years old.

Hello, my name is Peter and I am 41 years old.

Hello, my name is Peter1 and I am 21 years old.

Actually, I have a trap here. For example, I now call it in a different order (setAge and setName interchange for the first time), as follows:

Person (null, "kitty", 45) .setAge (41). SetName ("Peter"). SetName ("Peter1"). SetAge (21). Introduce

Then the result will look like this:

Hello, my name is kitty and I am 41 years old.

Hello, my name is Peter and I am 41 years old.

Hello, my name is Peter1 and I am 21 years old.

Third, sum up

The above two results occur because our setAge operation returns the object itself after execution, and the setName operation re-new an object.

From this, we can compare to the operation of RDD. Before, the group asked a question like this:

RDD.repartiton (12)

Why doesn't the partition of RDD become 12?

In fact, it is because all the transformation operators of RDD generate a new RDD, rather than acting the function on itself.

In fact, another way to implement chained computation is to execute a function that returns a fixed type, not necessarily the caller itself or the implementation object of the same parent class. For example, the final implementation function executed by Dataset returns a fixed type: RDD [InternalRow], not Dataset.

After reading the above, have you mastered how to combine demo and spark in Scala language to realize chained computing? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.