Shulou(Shulou.com)06/03 Report--

16-03-04 00:21:09 WARN SparkContext: Using SPARK_MEM to set amount of memory to use per executor process is deprecated, please use spark.executor.memory instead.

16-03-04 00:21:09 ERROR SparkContext: Error initializing SparkContext.

Org.apache.spark.SparkException: Could not parse Master URL: 'at'

At org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler (SparkContext.scala:2554)

At org.apache.spark.SparkContext. (SparkContext.scala:489)

At com.bigdata.deal.scala.DomainLib$.main (DomainLib.scala:22)

At com.bigdata.deal.scala.DomainLib.main (DomainLib.scala)

At sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod)

At sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)

At sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)

At java.lang.reflect.Method.invoke (Method.java:606)

At org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain (SparkSubmit.scala:664)

At org.apache.spark.deploy.SparkSubmit$.doRunMain$1 (SparkSubmit.scala:169)

At org.apache.spark.deploy.SparkSubmit$.submit (SparkSubmit.scala:192)

At org.apache.spark.deploy.SparkSubmit$.main (SparkSubmit.scala:111)

At org.apache.spark.deploy.SparkSubmit.main (SparkSubmit.scala)

When configuring conf, be sure to have an sparkHome master address

Note that you can configure these in spark-env.sh.

[root@mini-cp1 spark-1.4.0-bin-hadoop2.6] # cat conf/spark-env.sh

#! / usr/bin/env bash

SPARK_MASTER_IP=mini-cp1

# the JAVA root path must be imported

Export JAVA_HOME=/usr/local/jdk1.7.0_65

Export HADOOP_HOME=/usr/local/hadoop-2.6.0

# export SCALA_HOME=/opt/scala

Export SPARK_WORKER_MEMORY=3g

Export HADOOP_CONF_DIR=/usr/local/hadoop-2.6.0/etc/hadoop

# SPARK_MEM=$ {SPARK_MEM:-1g}

Export SPARK_MEM=3g

Export HADOOP_HOME=/usr/local/hadoop-2.6.0

Export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/hadoop-2.6.0/lib/native

Export HADOOP_OPTS= "- Djava.library.path=/usr/local/hadoop-2.6.0/lib"

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.