Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to build Hadoop cluster in Docker". In daily operation, I believe many people have doubts about how to build Hadoop cluster in Docker. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "how to build Hadoop cluster in Docker". Next, please follow the editor to study!

one。 Project introduction

By packaging Hadoop into a Docker image, you can quickly build a Hadoop cluster on a single machine, which makes it easy for beginners to test and learn.

As shown in the following figure, Hadoop's master and slave run in different Docker containers, with NameNode running in the hadoop-master container and DataNode and NodeManager running in the ResourceManager,hadoop-slave container. NameNode and DataNode are the components of Hadoop distributed file system HDFS, which are responsible for storing input and output data, while ResourceManager and NodeManager are components of Hadoop cluster resource management system YARN, which is responsible for the scheduling of CPU and memory resources.

Previous versions used serf/dnsmasq to provide DNS services for Hadoop clusters, but they are no longer needed due to updates to Docker network capabilities. In the newer version, use the following command to create a separate network for the Hadoop cluster:

Sudo docker network create-driver=bridge hadoop

Then when running the Hadoop container, use the "- net=hadoop" option, and all containers will run on the hadoop network, and they can communicate through the container name.

Key points of project update:

Remove serf/dnsmasq

Merge Master and Slave images

Install using Hadoo compiled by the kiwenlau/compile-hadoop project

Optimize Hadoop configuration

two。 Step 1 of building a 3-node Hadoop cluster. Download the Docker image sudo docker pull kiwenlau/hadoop:1.02. Download GitHub Warehouse git clone https://github.com/kiwenlau/hadoop-cluster-docker3. Create Hadoop network sudo docker network create-- driver=bridge hadoop4. Run the Docker container cd hadoop-cluster-docker./start-container.sh

Running result

Start hadoop-master container...start hadoop-slave1 container...start hadoop-slave2 container...root@hadoop-master:~#

3 containers, 1 master, 2 slave started

After running it, you will enter the / root directory of the hadoop-master container

5. Start hadoop./start-hadoop.sh6. Run wordcount./run-wordcount.sh

Running result

Input file1.txt:Hello Hadoopinput file2.txt:Hello Dockerwordcount output:Docker 1Hadoop 1Hello 2

Hadoop webpage management address:

NameNode: http://192.168.59.1:50070/

ResourceManager: http://192.168.59.1:8088/

192.168.59.1 is the IP of the host running the container.

three。 Step 1 of setting up N-node Hadoop cluster. Prepare for

Refer to part 2: download the Docker image, download the GitHub repository, and create a Hadoop network

two。 Rebuild the Docker image. / resize-cluster.sh 5

You can specify any N (N > 1)

3. Start the Docker container. / start-container.sh 5

Be consistent with N in step 2.

4. Run Hadoop

Refer to part 2, 5: 6: start Hadoop and run wordcount.

At this point, the study on "how to build a Hadoop cluster in Docker" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.