(version customization) lesson 15: No Receivers thorough thinking on the interpretation of Spark Streaming Source Code

Hu This issue content:

1. Kafka decryption

Background:

Currently No Receivers is used more and more in enterprises, and No Receivers has stronger control and semantic consistency. No Receivers is our natural way of manipulating data sources using an wrapper and is of type RDD.

So Spark Streaming produces custom RDD -> KafkaRDD.

Source code analysis:

1. KafkaRDD source code

private[kafka]class KafkaRDD[K: ClassTag,V: ClassTag,U val (host, port) = leaders(TopicAndPartition(o.topic, o.partition))new KafkaRDDPartition(i, o.topic, o.partition, o.fromOffset, o.untilOffset, host, port) }.toArray }

2、HasOffsetRanges

/** * Represents any object that has a collection of [[OffsetRange]]s. This can be used to access the * offset ranges in RDDs generated by the direct Kafka DStream (see * [[KafkaUtils.createDirectStream()]]). * {{{* KafkaUtils.createDirectStream(...). foreachRDD { rdd => * val offsetRanges = rdd.asInstanceOf[HasOffsetRanges].offsetRanges * ... * } * }}}*/trait HasOffsetRanges {def offsetRanges: Array[OffsetRange]}

3. compute in Kafka RDD

override def compute(thePart: Partition, context: TaskContext): Iterator[R] = {val part = thePart.asInstanceOf[KafkaRDDPartition]assert(part.fromOffset {offsetRange.topic}{offsetRange.partition}+{offsetRange.fromOffset}{offsetRange.untilOffset}}.mkString()metadata = (-> offsetRanges.toListStreamInputInfo.-> description)inputInfo = (rdd.countmetadata) ssc... reportInfo(validTimeinputInfo)= untilOffsets.map(kv => kv._ 1 -> kv._ 2.offset)(rdd)}

Benefits of Direct?

1. Direct mode has no data cache, so there is no memory overflow, but if you use Receiver, you need cache.

2. If Receiver mode is adopted, it is inconvenient to do distribution, while Direct mode defaults to data on multiple machines.

3. In actual operation, if the disadvantage of using Receiver is to assume that the data is too late to process, but Direct will not, because it reads the data directly.

4. Semantic consistency, the Direct way data must be enforced.