Shulou(Shulou.com)06/01 Report--

This article mainly explains "Spark configuration method sharing", interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let Xiaobian take you to learn "Spark configuration method sharing"!

1. Spark can be configured in three places: spark properties control most application parameters, via SparkConf objects or Java system properties; environment variables can be set individually on each machine via conf/spark-env.sh;Logging can be set via log4j.properties

2. SparkConf is set separately for each application. You can set masterURL, application name, and arbitrary key-value pairs through the corresponding set method. Spark-shell and spark-submit support two ways to load configuration dynamically, the first through command-line arguments and the second through the--conf flag./ bin/spark-submit --help to view command parameters. bin/spark-submit also loads configuration information from the conf/spark-defaults.conf file, which contains space-separated key-value pairs. The priority of configurations, from highest to lowest, is SparkConf > Parameters passed to spark-shell or spark-submit> spark-defaults.conf file.

3. Common application configuration parameters

spark.executor.memory

At this point, I believe that everyone has a deeper understanding of "spark configuration method sharing", may wish to actually operate it! Here is the website, more related content can enter the relevant channels for inquiry, pay attention to us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.