Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is the mapWithState decryption method of Spark". The content of the explanation is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "what is the mapWithState decryption method of Spark".

The updateStateByKey and mapWithState methods are not found in DStream and need to be implicitly converted to PairDStreamFunctions objects.

UpdateStateByKey is an update operation in an existing state. Historical data is stored in Seq [V], and new values are stored in Option [S], which may have no value. The default Partitioner function, HashPartitioner is used.

StateDStream inherits DStream and uses MEMORY_ONLY_SER as the storage level.

The compute method of StateDStream, which calls the computeUsingPreviousRDD method if parent exists, and the mapPartitions method that invokes preStateRDD if no one exists.

In the computeUsingPreviousRDD method, parentRDD and the previous RDD perform cogroup operations, aggregate the Value according to Key, and scan all Value based on Key. When the amount of data is small, you can consider using the cogroup method, but when the amount of data is large, it will affect the whole operation and reduce performance.

The parameter of type StateSpec is received in the mapWithState method, and the function of the update operation is encapsulated in StateSpec.

StateImpl inherits the State class, records its status information, and defines access, update, delete and other operation interfaces, much like a table in the database to add, delete, modify and check.

The essence of the compute method of the MapWithStateDStreamImpl class is to call the getOrCompute method of InternalMapWithStateDStream.

InternalMapWithStateDStream is an update operation based on historical data.

In the MapWithStateRDD class, one Partition corresponds to one MapWithStateRDDRecord.

The compute method of MapWithStateRDD, which first gets the previous State information, operates with the iterator of the current data, calls the updateRecordWithData method of MapWithStateRDDRecord, and returns the iterator with MapWithStateRDDRecord as the element.

The entire historical data is saved in newStateMap.

Thank you for your reading, the above is the content of "what is the mapWithState decryption method of Spark". After the study of this article, I believe you have a deeper understanding of what the mapWithState decryption method of Spark is, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.