Shulou(Shulou.com)06/01 Report--

This article is to share with you about how to configure spark in zeppelin, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Modify zeppelin configuration file

Add spark path and hadoop profile path to zeppelin-env.sh

Export SPARK_HOME=/usr/lib/sparkexport HADOOP_CONF_DIR=/usr/lib/hadoop modifies spark interpreter parameters

Master can be:

Local [*] in local modespark://master:7077 in standalone clusteryarn-client in Yarn client modeyarn-cluster in Yarn cluster modemesos://host:5050 in Mesos cluster

I am spark on yarn here, so the master I configured is yarn, the submission mode is cluster, and the running task is stuck when I set it to client.

Troubleshooting

I built this in docker, and I reported other mistakes in the middle.

Java.lang.RuntimeException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

It turned out to be the reason why AM could not be instantiated.

Host-I

Found that there is a virtual ip in docker

Solution:

Specify the network connection method when starting docker

Host mode: docker run uses-- net=host specifies that the network used by the docker is actually the same as the host. The above is how to configure spark in zeppelin. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.