Shulou(Shulou.com)06/01 Report--

This article introduces how to analyze the startup log of Spark cluster, which is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

Created by Wang, Jerry, last modified on Aug 24, 2015

Added by Jerry: …

/ root/devExpert/spark-1.4.1/sbin/... / conf-(Jerry: I haven't copied out the template from my own confiugration file yet)

Starting org.apache.spark.deploy.master.Master, logging to / root/devExpert/spark-1.4.1/sbin/... / logs/spark-root-org.apache.spark.deploy.master.Master-1-NKGV50849583FV1.out

Jerry- location of log file:

NKGV50849583FV1:~/devExpert/spark-1.4.1/logs # vi spark-root-org.apache.spark.deploy.master.Master-1-NKGV50849583FV1.out

1 added by Jerry: loading load-spark-env.sh! 1

2 added by Jerry, number of Jars: 1

3 added by Jerry, launch_classpath: / root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar

4 added by Jerry,RUNNER:/usr/jdk1.7.0_79/bin/java

5 added by Jerry, printf argument list: org.apache.spark.deploy.master.Master-ip NKGV50849583FV1-port 7077-webui-port 8080

Jerry: this is default value

6 Spark Command: / usr/jdk1.7.0_79/bin/java-cp / root/devExpert/spark-1.4.1/sbin/... / conf/:/root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assem bly-1.4.1-hadoop2.4.0.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucle us-core-3.2.10.jar:/root/devExpert/spark-1.4. 1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar-Xms512m-Xmx512m-XX:MaxPermSize=256m org.apache.spark.dep loy.master.Master-ip NKGV50849583FV1-port 7077-webui-port 8080

7 = =

8 added by Jerry, I am in if-else branch: / usr/jdk1.7.0_79/bin/java-cp / root/devExpert/spark-1.4.1/sbin/... / conf/:/root/devExpert/spark-1.4.1/assembly/targ et/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/root/devExpert/spark-1.4.1/l ib_managed/jars/datanucleus-core-3.2.10.jar:/root/devExpert/spark-1.4. 1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar-Xms512m-Xmx512m-XX:MaxPermSize= 256m org.apache.spark.deploy.master.Master-ip NKGV50849583FV1-port 7077-webui-port 8080

9 Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

10 15-08-16 12:45:17 INFO Master: Registered signal handlers for [TERM, HUP, INT]

11 12:45:17 on 15-08-16 WARN Utils: Your hostname, NKGV50849583FV1 resolves to a loopback address: 127.0.0.1; using 10.128.184.131 instead (on interface eth0)

12 15-08-16 12:45:17 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address

Jerry: useful hint

13 12:45:18 on 15-08-16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform … Using builtin-java classes where applicable

14 15-08-16 12:45:18 INFO SecurityManager: Changing view acls to: root

15 15-08-16 12:45:18 INFO SecurityManager: Changing modify acls to: root

16 12:45:18 on 15-08-16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set (root); users with mo dify permissions: Set (root)

17 15-08-16 12:45:19 INFO Slf4jLogger: Slf4jLogger started

18 15-08-16 12:45:19 INFO Remoting: Starting remoting

19 15-08-16 12:45:19 INFO Remoting: Remoting started; listening on addresses: [akka.tcp://sparkMaster@NKGV50849583FV1:7077]

20 12:45:19 on 15-08-16 INFO Utils: Successfully started service 'sparkMaster' on port 7077.

21 15-08-16 12:45:19 INFO Utils: Successfully started service on port 6066.

22 15-08-16 12:45:19 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066

23 15-08-16 12:45:19 INFO Master: Starting Spark master at spark://NKGV50849583FV1:7077

24 15-08-16 12:45:19 INFO Master: Running Spark version 1.4.1

25 12:45:19 on 15-08-16 INFO Utils: Successfully started service 'MasterUI' on port 8080.

26 12:45:19 on 15-08-16 INFO MasterWebUI: Started MasterWebUI at http://10.128.184.131:8080

27 15-08-16 12:45:20 INFO Master: I have been elected leader! New state: ALIVE-cool!

On how to analyze the startup log of the Spark cluster is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.