Shulou(Shulou.com)06/01 Report--

This article introduces the relevant knowledge of "how to install Hadoop". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

1. Set up a virtual machine with vmware workstation. Do not choose to install the package at first.

Memory selection: if the machine has 4G memory, allocate 1G memory to the virtual machine

2. Complete the preliminary configuration of the virtual machine according to the default requirements

[big data developed learning materials]: join big data Technical Learning Exchange Group 458345782, click to join the group chat, and the private message administrator can get it for free.

3. Start the installation of ubuntu

Select the image file (the version I use is also downloaded from the network disk above), and then install it step by step

4. Choose the computer name as hd1 and password as 123456. Notice that Your name is set to ubuntu

5. Open the Ubuntu virtual machine and the terminal editor

6. Install and enter VMware Tools. Install by clicking Virtual Machine-> install VMware Tools.

7. Copy the VMware Tools package to another file directory. Put it here in the tmp directory.

8. Decompress VMWare Tools

9. Enter vmware-tools-distrib after decompression

10. Continue to write commands to install sudo./vmware-install.pl

11. VMware Tools was installed successfully

12. Install vim using sudo aptinstall vim. After the installation is completed, it is shown below.

13. Type reboot to restart, and after restart, enter init 0 to shut down the virtual machine

14. Copy the hd1 folder, named hd2, and put it under the folder where the virtual machine is stored

15. Use notepad to open hd1.vmx under the hd2 folder. Change the displayName to hd2.

Then use VMware WorkStation to open hd1.vmx

16. Start two virtual machines respectively

17. According to the computer's ip configuration / etc/hosts file, give the IP address an alias to facilitate later operation. Note that the ip address is the ip address of your two virtual machines. You can view the native IP with the ifconfig command. It's usually 191.168. The first string of IP addresses. My configuration is as follows.

Sudo vim / etc/hosts

192.168.241.132 hd1

192.168.241.130 hd2

18. Install openssh-server

19. Start ssh

20. Make sure to start ssh automatically every time you turn it on.

21. Multiple computers trust each other, and both virtual machines have to execute ssh-keygen-t rsa commands.

22. For mutual trust of keys, both virtual machines need to execute the command cat id_rsa.pub > > authorized_keys.

23. Copy hd1 files to hd2 and hd2 files to hd1.

24. Verify that mutual trust between computers has been realized.

25. Install hadoop on hd1

26. Install java on hd1 and hd2.

Both hosts execute the sudo apt install default-jre command.

27. Create related directories on hd1 and hd2 respectively

Next, you need to configure the hadoop cluster file

There are 7 profiles:

/ home/ubuntu/hadoop/hadoop/etc/hadoop/hadoop-env.sh

/ home/ubuntu/hadoop/hadoop/etc/hadoop/yarn-env.sh

/ home/ubuntu/hadoop/hadoop/etc/hadoop/slaves

/ home/ubuntu/hadoop/hadoop/etc/hadoop/core-site.xml

/ home/ubuntu/hadoop/hadoop/etc/hadoop/hdfs-site.xml

/ home/ubuntu/hadoop/hadoop/etc/hadoop/mapred-site.xml

/ home/ubuntu/hadoop/hadoop/etc/hadoop/yarn-site.xml

28. Configure env file. Steps 28-34 are completed only on hd1

Export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/

Export HADOOP_HOME=/home/hadoop/hadoop

29. Configure the yarn-env file

Export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/

30. Configure slave file

Hd1

Hd2

31. Configure the core-site file

Fs.defaultFS

Hdfs://hd1:9000

Io.file.buffer.size

131072

Hadoop.tmp.dir

File:/home/ubuntu/hadoop/tmp

Hadoop.proxyuser.hduser.hosts

*

Hadoop.proxyuser.hduser.groups

*

32. Configure the hdfs-site file

Dfs.namenode.http-address

Hd1:50070

Dfs.namenode.secondary.http-address

Hd1:9001

Dfs.namenode.name.dir

File:/home/ubuntu/hadoop/namenode

Dfs.datanode.data.dir

File:/home/ubuntu/hadoop/hdfs

Dfs.replication

one

Dfs.webhdfs.enabled

True

Dfs.support.append

True

Dfs.support.broken.append

True

Dfs.permissions.enabled

False

33. Configure the mapred-site file

Be careful to copy out the mapred-site.xml.template file before using the vim command

Mapreduce.framework.name

Yarn

Mapreduce.jobhistory.address

Hd1:10020

Mapreduce.jobhistory.webapp.address

Hd1:19888

[big data developed learning materials]: join big data Technical Learning Exchange Group 458345782, click to join the group chat, and the private message administrator can get it for free.

34. Configure the yarn-site file

Yarn.nodemanager.aux-services

Mapreduce_shuffle

Yarn.nodemanager.aux-services.mapreduce.shuffle.class

Org.apache.hadoop.mapred.ShuffleHandler

Yarn.resourcemanager.address

Hd1:8032

Yarn.resourcemanager.scheduler.address

Hd1:8030

Yarn.resourcemanager.resource-tracker.address

Hd1:8031

Yarn.resourcemanager.admin.address

Hd1:8033

Yarn.resourcemanager.webapp.address

Hd1:8088

35. Configure path files on hd1 and hd2. Execute the following two commands

Sudo vim / etc/profile

Export PATH=$PATH:/home/ubuntu/hadoop/hadoop/bin:/home/ubuntu/hadoop/hadoop/sbin

36. Add the hadoop path to the path path on hd1

37. Format namenode on hd1 and execute the hadoop namenode-format command

38. Distribute the files on the hd1, paying attention to the current location of the command line.

39. Verify that the installation is successful

Create an a.txt file on hd1

Copy it into the hdfs file

Print display

Since then the installation has been successful

This is the end of "how to install Hadoop". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.