Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build a Hadoop cluster environment for ubuntu docker. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Spark should be used with Hadoop's hdfs. However, Hadoop is distributed, so it is difficult to build a cluster on a host. Baidu found that it could be built using docker, so it started:

Github Project: https://github.com/kiwenlau/hadoop-cluster-docker

Docker installation

Docker.io is installed in the article

However, I recommend that the docker-ce,docker.io version is too old. The steps are as follows:

1. The source of APT software package under the update of international practice.

Sudo apt-get update

2. Install the software package to allow apt to use the repository through HTTPS

Sudo apt-get install\ apt-transport-https\ ca-certificates\ curl\ software-properties-common

3. Because the education network image is used by the wall:

The copy code is as follows:

Curl-fsSL https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu/gpg | sudo apt-key add

4. Similarly

Sudo add-apt-repository\ "deb [arch=amd64] https://mirrors.ustc.edu.cn/docker-ce/linux/ubuntu\ $(lsb_release-cs)\ stable"

5. Update our software source

Sudo apt-get update

6. Install docker

Sudo apt-get install docker-ce

7. Start the docker service

Sudo service docker start

Or

Sudo systemctl start docker

8. If you feel it is necessary, you can set up the boot and bring your own boot.

Sudo systemctl enable docker

9. Turn off self-start

Sudo systemctl disable docker

10. Domestic access to docker will be restricted, and acceleration services such as Aliyun, NetEase Cloud and DaoCloud can be used.

11. Run docker run hello-world test to see if the installation is successful

12. If you want to install other image, you can find it on docker hub

Https://hub.docker.com/

Or search for the corresponding docker directly on github

Set up Hadoop cluster

The building process is simple.

Https://github.com/kiwenlau/hadoop-cluster-docker

The instructions in the previous steps are very detailed:

1. Pull image, if the speed is slow, change the domestic image source.

Sudo docker pull kiwenlau/hadoop:1.0

More than 200m is not big, I have played jupyter official pyspark docker 5g …

2. Clone the project locally

Git clone https://github.com/kiwenlau/hadoop-cluster-docker

In fact, only the start-container.sh files inside can be copied and pasted separately.

3. Start-container.sh needs to modify cd to the file directory

The sudo gedit start-container.sh is modified as follows

Open port 9000 and create shared folders for later use

4. After saving, create a docker-Hadoop network

Sudo docker network create-driver=bridge hadoop

5. Open the container

Sudo. / start-container.sh

6. Start the Hadoop cluster

. / start-hadoop.sh

7. Test Hadoop and create a test/input directory on hdfs

Hadoop fs-mkdir-p / test/inputhadoop fs-ls / test

8. Run word-count program

. / run-wordcount.sh

The results are as follows

This is the end of the article on "how to build a Hadoop cluster environment for ubuntu docker". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.