In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-04-03 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >
Share
Shulou(Shulou.com)06/03 Report--
1. Principle of reverse pressure mechanism
The reverse pressure mechanism in Spark Streaming is a new feature introduced by Spark 1.5.0, which can dynamically adjust the intake rate according to the processing efficiency.
When the batch processing time (Batch Processing Time) is greater than the batch interval (Batch Interval, that is, BatchDuration), it means that the speed of data processing is less than the speed of data intake, and the duration is too long or the source data increases sharply, which is easy to cause data to accumulate in memory and eventually lead to Executor OOM or task collapse.
In this case, if it is a Kafka Receiver-based data source, you can control the maximum input rate by setting spark.streaming.receiver.maxRate; if it is a Direct-based data source (such as Kafka Direct Stream), you can control the maximum input rate by setting spark.streaming.kafka.maxRatePerPartition. Of course, it is generally not a problem to set these parameters when the pressure test has been carried out in advance and the flow peak will not exceed expectations. However, the maximum value does not mean that it is the optimal value, and it is best to dynamically estimate the optimal rate of the next batch according to the processing of each batch. Above Spark 1.5.0, it can be realized by back pressure mechanism. Turn on the reverse pressure mechanism, that is, setting spark.streaming.backpressure.enabled to true,Spark Streaming will automatically adjust the input rate according to the processing capacity, so as to ensure maximum throughput and performance at peak traffic.
Override def onBatchCompleted (batchCompleted: StreamingListenerBatchCompleted) {val elements = batchCompleted.batchInfo.streamIdToInputInfo for {/ / processing end time processingEnd
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.