Shulou(Shulou.com)06/01 Report--

This article is to share with you what the advantages of Spark are, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

When using Spark to process data, if the amount of data is small, then the default configuration of Spark can basically meet the actual business scenarios. However, when the amount of data is large, it is necessary to adjust and optimize the parameter configuration to ensure the safe and stable operation of the business. And in the actual optimization, different scenarios should be considered and different optimization strategies should be adopted. 1. Reasonable setting of micro-batch processing time in SparkSreaming flow processing, reasonable setting of micro-batch processing time (batchDuration) is very necessary. If the batchDuration setting is too short, it will cause SparkStreaming to submit job frequently. If the job generated by each batchDuration cannot be processed within this time, it will cause the job to accumulate continuously, resulting in SparkStreaming blocking and even program downtime. It needs to be determined according to different application scenarios and hardware configurations, and the batchDuration can be adjusted according to the visual monitoring interface of SparkStreaming and observing Total Delay and other indicators. two。 To control the maximum rate of consumption, such as SparkStreaming and Kafka integration, when using direct mode, you need to set the parameter spark.streaming.kafka.maxRatePerPartition to control the maximum consumption per Kafka partition. This parameter is not online by default, that is, it will pull out all the data in the Kafka directly. However, in practical use, it needs to be considered comprehensively according to the rate at which producers write Kafka and the speed at which consumers process data. At the same time, we also need to combine the above batchDuration to ensure that the data pulled by each partition should be successfully processed during each batchDuration to achieve the highest possible throughput. The adjustment of this parameter needs to refer to the Input Rate and Processing Time in the visual monitoring interface. 3. Cache repeatedly used "dataset" RDD in Spark and DStream in SparkStreaming, if used repeatedly, it is best to use cache or persist operator to cache the "dataset" to prevent unnecessary overhead caused by excessive scheduling resources. 4. Reasonable setting of GCJVM garbage collection is very time-consuming and performance-consuming, especially stop world and full gc affect the normal operation of the program. For JVM and parameter configuration, it is recommended to study "JVM memory management and garbage collection", "JVM garbage collector, memory allocation and collection strategy", "memory leaks, memory spills and out-of-heap memory, JVM optimized configuration parameters". 5. A reasonable setting of CPU can occupy one or more core for each executor. You can understand the usage of computing resources by observing the changes in the utilization of CPU. To avoid wasteful use of CPU, for example, one executor takes up multiple core, but the overall CPU utilization is not high. At this point, it is recommended that each executor take up less core than before, and more executor processes are added under the worker to increase the number of executor executed in parallel, thus improving CPU utilization. At the same time, memory consumption should be taken into account. After all, the more executor a machine is running, the smaller the memory of each executor is, and it is easy to generate OOM. 6. Serialization and deserialization using Kryo Spark defaults to Java's serialization mechanism, but this Java native serialization mechanism performs much worse than Kryo. To use Kryo, you need to set the serializer to KryoSerializerSparkConf.set ("spark.serializer", "org.apache.spark.serializer.KryoSerializer") / / register the custom type SparkConf.registerKryoClasses to serialize (Array (classOf [CustomClass1], classOf [CustomClass2]))

7. Use high-performance operator 1) use reduceByKey, aggregateByKey instead of groupByKey2) filter after coalesce operation

3) use repartitionAndSortWithinPartition

Replace repartition and sort operation

4) use mapPartition instead of map

5) use foreachPartition instead of foreach

The operator substitution optimization should be carried out according to the actual use scene.

These are the advantages of Spark. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.