Shulou(Shulou.com)06/02 Report--

1. Big data and Hadoop

To study and study big data, it is natural to start with Hadoop. Hadoop is not a simple software, but a series of ecology of software formation. Its core idea comes from three papers originally published by Google, and later made an open source implementation. Google's implementation and Hadoop's implementation can roughly correspond to this:

Google Map/Reduce Hadoop MapReduce Google GFS Hadoop HDFS Google BigTable Hadoop HBase

All the components of Hadoop ecology run in the linux environment, so naturally we need to build the linux environment first. In addition, Hadoop can deal with "large" data because of its distributed characteristics, it can use distributed computing to build server clusters, and can be expanded as needed. In order to learn, we first use virtual machines on the local computer to build a linux environment. To model the cluster environment, just create a few more virtual machines.

In order to play with this set of things, because you have to create several virtual machines, your computer had better have no less than 8 gigabytes of memory, because running Hadoop generally requires 2 gigabytes of virtual machine, and opening three virtual machines at random takes up a lot of memory. However, in order to learn in the early stage, I only allocated 1G of memory to each virtual machine, and then it was not enough to adjust.

two。 Why VirtualBox?

When it comes to virtual machines, the first thing that comes to mind is the famous VMware. VMware is an established virtual machine software, and there are many online tutorials. I chose VirtualBox mainly because of the problem of license. I only have one MacBook pro on hand, so I don't want to spend too much time on the software license, so I chose the free VirtualBox. The installation process of VirtualBox is not impressive, but it is a routine operation, not to mention it.

3. Install CentOS in the virtual machine

As mentioned earlier, my main computer is a MacBook Pro. For the previous aspect, a virtual Windows was created using Parallel Desktop, which is not mentioned here.

Linux system I chose CentOS, first download the ISO file of the system from the official website of CentOS. Create a new virtual machine in VirtualBox, select Linux as the type, and choose any version as you like, because there is no CentOS in it, but it's okay to choose one. Note that it is best to choose the folder, which is where the virtual machine files are stored, because as more and more things are installed, the virtual machine files may become larger and larger. My computer hard drive is almost full, so I hung an external hard disk and put the virtual machine files on the external hard disk and plugged in when needed. Although it was inconvenient, I had to make do with it.

Other parameters can be set by default. After the virtual machine is created successfully, click to set it, switch to storage, and select the downloaded ISO file of the CentOS system in the CD-ROM drive, so that the virtual machine can enter the installation of CentOS after startup.

Install the CentOS7 operating system not to mention, it is a graphical interface, it is not difficult.

4. Potholes encountered in network environment configuration

The network setup after the successful installation of Linux needs to be mentioned here, I stepped on a lot of holes in this. I hope my network environment is like this. First of all, we want the Linux virtual machine to be connected to the Internet, so that it will be easier to download what components we need to download in the future. Second, I want to be able to access the Linux virtual machine from the Host machine, that is, MacBook, so that ssh can log in. The default small window of VirtualBox is really small, and you can adjust it, although it can be adjusted, but it is more convenient to operate under the host, so you still need to ssh it from MacBook. Please note, however, that CentOS cannot be connected to the Internet after it has just been installed, and relevant settings need to be made.

4.1 Linux virtual machine to connect to the Internet

The network setting of the virtual machine can choose "Network address Translation NAT" by default, and the virtual machine creates a NAT network.

After the virtual machine is installed, the network is not enabled by default, so it cannot be connected to the network. You need to start the virtual machine and change the network configuration after logging in.

Vi / etc/sysconfig/network-scripts/ifcfg-enp0s3

Change ONBOOT to yes so that linux starts the network as soon as it starts. Press ESC after the change, and then: wq to exit the save. Instead of restarting, restart the network service with the following command for the changes to take effect.

Systemctl restart network

At this point, you can check it with the command ```ip addr```. You will see that the network already has an IP address, for example, mine is 10.0.2.15. As shown in the figure:

Now if you ping www.baidu.com, you should find that you have been able to connect to the Internet.

4.2 access the Linux virtual machine from the host Macbook

In addition, from the host Macbook still can not ping this address, that is, the host can not access the virtual machine. For the host MacBook and the virtual machine linux to interconnect, you need to keep the two machines on the same network segment. In VirtualBox, under the "manage" menu, open "Host Network Manager" and create one, as shown in the figure. You can see that a virtual network card of 192.168.56.1 has been created.

At this time, if you use the ifconfig command in the MacBook terminal to check, you will find that there is an extra vboxnet0 network card, and the ip address is 192.168.56.1.

Danieldu@daniels-MacBook-Pro-857  ~  ifconfig lo0: flags=8049 mtu 16384 options=1203 inet 127.0.0.1 netmask 0xff000000 inet6:: 1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 nd6 options=201 gif0: flags=8010 mtu 1280 stf0: flags=0 mtu 1280 en0: flags=8863 mtu 1500 ether ac:bc:32:c1:ed:dd inet6 fe80::1c82:47a:64f:460f%en0 prefixlen 64 secured scopeid 0x4 inet 192.168.31.46 netmask 0xffffff00 broadcast 192.168.31.255 nd6 options=201 media: autoselect status: active.

Vboxnet0: flags=8943 mtu 1500 ether 0a:00:27:00:00:00 inet 192.168.56.1 netmask 0xffffff00 broadcast 192.168.56.255 utun1: flags=8051 mtu 1380 inet6 fe80::f655:9c6f:ca10:240e%utun1 prefixlen 64 scopeid 0xc nd6 options=201

...

Then you need to add a network card 2. 5 to the corresponding virtual machine settings in VirtualBox. Select Host-Only Network only, and the interface name is the vboxnet0 you saw earlier.

Then log in to the virtual machine, use the "ip addr" command to check, and you will find that there is an extra network card enp0s8. Copy one from enp0s3, then edit the file, this time to a fixed IP address.

Cp / etc/sysconfig/network-scripts/ifcfg-enp0s3/etc/sysconfig/network-scripts/ifconfig-enp0s8

Then restart the network systemctl restart network. At this point the virtual machine can be accessed from the host Macbook.

4.3 turn off Linux Firewall

From the above settings, we can see that the IP of host MacBook is 192.168.56.1, and the static address of virtual machine Linux is 192.168.56.100. Now that it is in a network segment, it should be connected to ping. If you can't get through ping, it's probably a firewall problem. First of all, MacBook should turn off the firewall.

Then make sure that the virtual machine Linux also turns off the firewall, which is on by default.

Check the ssh service. It should be on by default.

You should now be able to log in to linux through ssh from mac's terminal.

Summary

The above is the method of using VirtualBox to build a local virtual machine environment on Mac. I hope it will be helpful to you. If you have any questions, please leave me a message and the editor will reply you in time. Thank you very much for your support to the website! If you think this article is helpful to you, you are welcome to reprint it, please indicate the source, thank you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.