Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "how to run Jupyter/Spark/Mesos service in Docker". Many people will encounter such a dilemma in the operation of actual cases, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Run the Jupyter/Spark/Mesos service in Docker.

Spark on Docker, based on Jupyter Notebook Python, Scala, R, Spark, Mesos technology stack, provides a remote operation model and task writing Web interface, using the Python world-famous Ipython Notebook format, very concise and friendly.

Integrated software

Jupyter Notebook 4.2.x

Conda Python 3.x and Python 2.7.x environment

Conda R 3.2.x environment

Scala 2.10.x

Pyspark, pandas, matplotlib, scipy, seaborn, scikit-learn are pre-installed in the Python environment

Ggplot2, rcurl originally installed in R environment

Spark 1.6.0, running in local mode, or connected to a Spark workers cluster

Mesos client 0.22 binary that can communicate with a Mesos master

Non-private username jovyan (uid=1000, configurable, see the corresponding option) in the group users (gid=100), with control in the directories / home/jovyan and / opt/conda

Tini as the entry point of the container and start-notebook.sh as the startup command

The script start-singleuser.sh, as an optional command, runs the single-user instance of Notebook server, which is required by JupyterHub

Options for HTTPS, password auth, and passwordless sudo

Usage

Use the following command to start a container, the Web service on port 8888, for configuration authorization (private network only, not on the Internet and other public networks).

Docker run-d-p 8888 8888 jupyter/all-spark-notebook

In general, you need to access the data resources in the host and use-v host-path:docker-path mapping.

After startup, type: http://127.0.0.1:8888 in the browser to access it.

Start Spark:Local Mode

Configuration using Spark in a small local data environment.

In Python Notebook:

Run a container, like above.

Open a Python 2 or 3 notebook.

Create SparkContext in local mode.

For example, in the first cell of notebook, it looks like this:

Import pysparksc = pyspark.SparkContext ('local [*]') # do something to prove it worksrdd = sc.parallelize (range (1000)) rdd.takeSample (False, 5) in R Notebook:

Run a container, like above.

Open an R notebook.

Initialize sparkR, in local mode.

Initialize sparkRSQL.

For example, in the first cell of R notebook, it looks like this:

Library (SparkR) sc

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.