Shulou(Shulou.com)05/31 Report--

This article mainly explains "what is RateController in Spark Streaming". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn "what is RateController in Spark Streaming"?

The model used in Spark Streaming is as follows, the basic idea is to send data to Controller according to the batch of 100ms. At the beginning, Controller is directly transferred to JobGenerator,JobGenerator to generate Job, which is processed by JobProcessor. JobProcessor sends job statistics to Controller,Controller and receives statistics, and dynamically changes Batch Size to send data to Job.

The subclasses of RateController are ReceiverRateController and DirectKafkaRateController.

ReceiverRateController sends a UpdateReceiverRateLimit message to receiverTracker.

After ReceiverTracker receives the UpdateReceiverRateLimit message, it forwards the UpdateRateLimit message to ReceiverSupervisorImpl.

After ReceiverSupervisorImpl receives the UpdateRateLimit message, it calls the updateRate method of BlockGenerator to control the receiving speed of the data.

BlockGenerator inherits from RateLimiter.

At this point, I believe you have a deeper understanding of "what is RateController in Spark Streaming". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.