(version customization) lesson 14: updateStateByKey and mapWithState decryption of State Management for Spark Streaming Source Code interpretation

Content of this issue:

1. updateStateByKey decryption

2. MapWithState decryption

Background:

Spark Streaming is divided into jobs according to Batch durations. But many times we need to calculate the data of the past day or even a week. At this time, it is inevitable to carry out state management, and Spark Streaming will generate a Job for each Batch Duration, which is RDD.

So the question at this point is how to maintain the state? At this point, you need to complete the core steps with the updateStateByKey and mapWithState methods.

1. Take a simple look at the updateStateByKey source code:

UpdateStateByKey and mapWithState in DStream are done by implicit conversion, there is no such method per se.

implicit def toPairDStreamFunctions[K, V](stream: DStream[(K, V)]) (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairDStreamFunctions[K, V] = {new PairDStreamFunctions[K, V](stream)}[: ClassTag]( updateFunc: ([]Option[]) => Option[] ): DStream[()] = ssc.withScope { updateStateByKey(updateFuncdefaultPartitioner())}

Eventually, this function will be accomplished through computeUsingPreviousRDD and compute in StateDStream. The simple flowchart is as follows:

2, a simple look at mapWithState source code

mapWithState is to return MapWithStateDStream object, maintain and update historical state is based on Key, use a function to maintain the state of key-value data

[: ClassTag: ClassTag]( spec: StateSpec[] ): MapWithStateDStream[] = {MapWithStateDStreamImpl[]( selfspec.asInstanceOf[StateSpecImpl[]] )}

This is done via compute in the InternalMapWithStateDStream class. The simple flowchart is as follows: