Shulou(Shulou.com)05/31 Report--

This article focuses on "what is Driver fault-tolerant security". Interested friends may wish to take a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what Driver fault-tolerant security is.

At the data level, ReceivedBlockTracker records metadata information for the entire Spark Streaming application.

From the scheduling level, DStreamGraph and JobGenerator are the core of Spark Streaming scheduling, recording the current scheduling progress, which is related to the business.

After receiving the metadata information, ReceivedBlockTracker calls the addBlock method, which is first written to disk and then written to memory.

According to the batchTime, the data to be processed by the current BatchDuration is allocated to the timToAllocatedBlocks data structure.

Time class is a case class, recording time, overloaded operators, implicit conversion, it is worth using for reference.

Case class Time (privateval millis: Long) {

Def milliseconds: Long = millis

Def

< (that: Time): Boolean = (this.millis < that.millis) def that.millis) def >

= (that: Time): Boolean = (this.millis > = that.millis)

Def + (that: Duration): Time = new Time (millis + that.milliseconds)

Def-(that: Time): Duration = new Duration (millis-that.millis)

Def-(that: Duration): Time = new Time (millis-that.milliseconds)

/ / Java-friendlier versions of the above.

Def less (that: Time): Boolean = this

< that def lessEq(that: Time): Boolean = this that def greaterEq(that: Time): Boolean = this >

= that

Def plus (that: Duration): Time = this + that

Def minus (that: Time): Duration = this-that

Def minus (that: Duration): Time = this-that

Def floor (that: Duration): Time = {

Val t = that.milliseconds

New Time ((this.millis / t) * t)

}

Def floor (that: Duration, zeroTime: Time): Time = {

Val t = that.milliseconds

New Time ((this.millis-zeroTime.milliseconds) / t) * t + zeroTime.milliseconds)

}

Def isMultipleOf (that: Duration): Boolean =

(this.millis% that.milliseconds = = 0)

Def min (that: Time): Time = if (this

< that) this else that def max(that: Time): Time = if (this >

That) this else that

Def until (that: Time, interval: Duration): Seq [Time] = {

(this.milliseconds) until (that.milliseconds) by (interval.milliseconds) map (new Time (_)

}

Def to (that: Time, interval: Duration): Seq [Time] = {

(this.milliseconds) to (that.milliseconds) by (interval.milliseconds) map (new Time (_)

}

Override def toString: String = (millis.toString + "ms")

}

Object Time {

Implicit val ordering = Ordering.by ((time: Time) = > time.millis)

}

Tracking the Time object, the input parameter batchTime in the allocateBlocksToBatch method of ReceiverTracker is called by the generateJobs method of JobGenerator.

The generateJobs method of JobGenerator is called by a timer to send a GenerateJobs message.

The time parameter in GenerateJobs is nextTime, and nextTime+=period, this period is ssc.graph.batchDuration.milliseconds.

The initial value of nextTime is assigned by the startTime passed in the start method, that is, the return value of the getStartTime method of RecurringTimer, which is the current time period (integer multiple + 1).

The value Period is the Duration value passed in when we call new StreamingContext to construct the StreamingContext.

ReceivedBlockTracker clears out-of-date metadata information and removes it from HashMap, which is also written to disk first and then to memory.

Metadata generation, consumption, and destruction all have WAL, so they can be recovered from the log if they fail. From the source code analysis, it is concluded that only when the checkpoint directory is set up can the WAL mechanism be carried out.

WAL the incoming checkpoint directory to create the log directory.

Here is to create a folder named receivedBlockMetadata under the checkpoint directory to hold the data recorded by WAL.

Checkpoint the state of the current DStream and JobGenerator, which is called at the end of the generateJobs method by sending a DoCheckpoint message.

At this point, I believe you have a deeper understanding of "what is Driver fault-tolerant security". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.