Shulou(Shulou.com)06/02 Report--

Today, I will talk to you about how to specify Driver and Executor in Spark1 jobs to use ports within the specified range. Many people may not know much about it. In order to make you understand better, the editor has summarized the following content for you. I hope you can get something according to this article.

1. Purpose of document writing

When submitting Spark jobs in CDH clusters, we all know that the communication ports between Driver and Executor of Spark are random, and Spark will choose the ports between 1024 and 65535 (inclusive), so it is not recommended to enable firewalls between clusters. The editor mainly introduces how to specify that Driver and Executor in Spark1 jobs use ports within the specified range for communication.

Content Overview

1. Configure Spark Driver and Executor port ran

two。 Verify port assignment

Test environment

1. Version 5.15 for CM and CDH

2. Spark version 1.6.0

two。 Configure Spark Driver and Executor port ran

1. Log in to the CM management interface and enter the Spark service configuration interface

two。 The configuration in the Gateway category also searches for "spark-defaults.conf" and adds the following configuration:

Spark.driver.port=10000

Spark.blockManager.port=20000

Spark.executor.port=30000

Spark.port.maxRetries=999

(can slide left and right)

3. Save the configuration and redeploy the client configuration for Spark2

3. Verify port assignment

1. Submit a Spark job to the cluster

Spark-submit-- class org.apache.spark.examples.SparkPi\

-- master yarn-client-- num-executors 4-- driver-memory 2g\

-- driver-cores 1-- executor-memory 2g-- executor-cores 1\

/ opt/cloudera/parcels/CDH/lib/spark/lib/spark-examples.jar 10

(can slide left and right)

The port 10000 here is mainly used to listen to Executor requests. When Executor is up, you need to communicate with Driver and obtain specific task information. It is the management segment call port used by Driver.

two。 Check the running interface of the Spark job to see the port numbers used by Driver and Executor

Here the ports of Driver and Executor are controlled by the parameter spark.blockManager.port, mainly to control the data transmission port between Driver and Executor.

3. View the startup log for each Executor

4. Summary

Port 10000 of 1.spark.driver.port is used to listen for requests from executor. When executor is up, it needs to communicate with driver and obtain specific task information. It is a management and scheduling port used by driver.

2.spark.blockManager.port (20000) ports are driver and executor direct data transfer ports (such as cached data frame, broadcast vars).

3. The spark.executor.port (30000) port is mainly used to interact with spark.driver.prot.

4. While Spark is running, blockManager will not interact with YARN, while driver will interact with Application Master processes running in YARN.

Tip: the code block part can be viewed by swiping left and right.

After reading the above, do you have any further understanding of how to specify Driver and Executor to use specified range ports in Spark1 jobs? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.