Shulou(Shulou.com)06/03 Report--

When developing Spark Streaming applications, it is necessary to improve the real-time performance of data processing as much as possible according to the configuration of each node in the cluster. In the process of tuning, on the one hand, we should make use of cluster resources as much as possible to reduce the time of each batch processing; on the other hand, we should ensure that the received data can be processed in time.

Run time optimization

Set reasonable batch time and window size

There is usually a dependency relationship between jobs in Spark Streaming, and the subsequent jobs must ensure that the previous jobs can only be submitted after the execution of the previous jobs. If the execution time of the previous jobs exceeds the set batch interval, then the subsequent jobs will not be submitted for execution on time, resulting in job jams. That is, if Spark Streaming applications are to run stably in the cluster, the received data must be disposed of as soon as possible. For example, if the batch processing time is set to 1 second, then the system generates a RDD every 1 second. If the system calculates a RDD for more than 1 second, then the current RDD has not yet had time to process, and the subsequent RDD has been submitted for processing, resulting in congestion. Therefore, you need to set a reasonable batch interval to ensure that the job can end within this batch interval. Many experimental data show that 500ms is a good batch processing interval for most Spark Streaming applications.

Similarly, for window operations, the sliding interval has a significant impact on performance. When the calculation cost of single batch data is too high, we can consider increasing the sliding time interval appropriately.

There is no uniform standard for setting batch time and window size. It usually starts with a larger batch time (about 10 seconds), and then constantly uses smaller values for comparison tests. If the processing time displayed in the Spark Streaming user interface remains the same, you can further set a smaller value; if the processing time begins to increase, it may have reached the application limit, and decreasing this value may affect the performance of the system.

Improve parallelism

Increasing parallelism is also a common way to reduce the time consumed by batch processing. There are three ways to improve parallelism. One way is to increase the number of receivers. If too much data is obtained, a single node may not have time to read and distribute the data, which makes the receiver become the bottleneck of the system. At this point, you can increase the number of receivers by creating multiple input DStream, and then use union to merge the data into one data source. The second method is to explicitly repartition the received data. If the number of receivers cannot be increased any more, the Dstream can be explicitly repartitioned by using parameters such as DStream.repartition, spark.streaming.blocklnterval, and so on. The third method is to improve the parallelism of aggregate computing. For operations that lead to shuffle, such as reduceByKey, reduceByKeyAndWindow, and so on, you can ensure that cluster resources are more fully utilized by setting higher row parameters through the display.

Memory usage and garbage collection

Control the amount of data during the batch interval

Spark Streaming stores all data obtained during the batch interval in the memory available within the Spark. Therefore, you must ensure that the amount of memory available to SparkStreaming on the current node can hold at least all the data in one batch interval. For example, if a batch interval is 1 second, but 1 second produces 1GB data, make sure that there is at least 1GB memory available for SparkStreaming on the current node.

Clean up data that is no longer used in a timely manner

Data that has been processed in memory and is no longer needed should be cleaned in a timely manner to ensure that Spark Streaming has enough memory space to use. One way is to clean up time-out useless data in time by setting a reasonable length of spark.cleaner.ttl, but this method should be used with caution to prevent subsequent data from being incorrectly cleaned up when needed. Another way is to set spark.streaming.unpersist to true, and the system will automatically clean up the RDD that is no longer needed. This method can significantly reduce the memory requirement of RDD and potentially improve the performance of GC. In addition, users can set a longer retention time for data by configuring the parameter streamingContext.remember.

Reduce the burden of serialization and deserialization

SparkStreaming serializes the received data into memory by default to reduce memory usage. Serialization and deserialization require more CPU resources, so CPU can be used more efficiently with appropriate serialization tools (such as Kryo) and custom serialization interfaces. In addition to using better serialization tools, you can also combine the compression mechanism to configure spark.rdd.compress in exchange for memory resources with the time overhead of CPU, reducing GC overhead.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.