Shulou(Shulou.com)05/31 Report--

This article shows you how to analyze the dynamic allocation of resources in Spark Job. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

The thrift-server of spark is used to do hdfs data query and statistics service in the project. When there is too much data, thrift-server needs to allocate a lot of resources, and it is very wasteful to allocate equivalent resources when there is no need for query, so think about whether it is possible to allocate resources dynamically.

(thrift-server is also a job (yarn app), so this method is also applicable to general spark job).

Currently, dynamic resource allocation is only suitable for spark on yarn, and the configuration is as follows.

Yarn node manager configuration

Modify the yarn node-manager configuration of all nodes and add to $HADOOP_HOME/etc/hadoop/yarn-site.xml:

Yarn.nodemanager.aux-services.spark_shuffle.classorg.apache.spark.network.yarn.YarnShuffleServicespark.shuffle.service.port7337

Add $SPARK_HOME/lib/spark-1.3.0-yarn-shuffle.jar copy to the $HADOOP_HOME/share/hadoop/yarn/lib directory.

Restart yarn.

Spark-default configuration

Modify the $SPARK_HOME/conf/spark-default.conf file to add the following:

Spark.shuffle.service.enabled truespark.shuffle.service.port 7337 spark.dynamicAllocation.enabled truespark.dynamicAllocation.minExecutors 1spark.dynamicAllocation.maxExecutors 12spark.dynamicAllocation.schedulerBacklogTimeout 1spark.dynamicAllocation.sustainedSchedulerBacklogTimeout 5spark.dynamicAllocation.cachedExecutorIdleTimeout 60 Verification

Thrift-server authentication built into spark is used here.

Start thrift-server and configure only the-- executor-memory parameter.

Sbin/start-thriftserver.sh-master yarn-client-executor-memory 1g

You can see the executors allocation through application UI. When there are concurrent queries or when the query pressure is high, more executor will be applied for and reclaimed when available.

The above content is how to analyze the dynamic allocation of resources in Spark Job. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.