Shulou(Shulou.com)06/03 Report--

As Hadoop needs to run in the Linux environment, and is distributed, so personal learning can only install virtual machines, this article is based on VMware Workstation, install CentOS7, the specific installation here is not too much introduction, only for the knowledge that needs to be used.

VMware installation, install a virtual machine and then use the way to copy the virtual machine to create the following virtual machines, saving time and effort, need to pay attention to the need to modify the IP and host name of each virtual machine.

All virtual machines use NAT mode to access the Internet, and ensure that they can access each other with the IP of the physical host.

There are several issues that need to be paid attention to. Nat if you want to surf the Internet, you first need to check the physical machine (PC). This server has been started. The principle of surfing the Internet: the traffic is the physical network card, but its ip has nothing to do with the physical machine. In other words, no matter how you change the network environment, your virtual machine does not need to change. This is its advantage.

First of all, after installing VMware, we will see two more network cards

If you go through the torture of network configuration, you should be able to find two network cards: vmnet1 and vmnet8. Here we are talking about vmnet8.

Focusing on the third picture on the right, you will see that the ip address, this ip address, varies from person to person, that is to say, different networks and different environments, the probability of this ip changing is very large. It could be 192.168.0.1 and so on. This is also equivalent to a variable of our programming, which requires you to assign values according to your environment. It is recommended that vmnet1 set 192.168.10.1 to 192.168.80.1 respectively to make a distinction.

Above we see the ip of the virtual machine network card, in order to avoid mixing, the virtual network card and the local network adapter are in the same interface. As shown below:

The following is after turning on the virtual machine, pop up through the menu

After popping up, we take a look at the virtual machine's (switch) vmnet8.

You can see that the vmnet8 subnet ip and the virtual machine Nic are from the same network segment.

At the same time, in the net settings interface on the right, we can see that the gateway is 192.168.106.2, which will be used if the virtual machine is changed to 192.168.80.2.

Basic knowledge of Linux

Learning Linux is also a process, because you may not even know the simplest boot and shutdown commands, let alone configure the network. This provides you with the materials and experience summary that you have just started to learn.

First of all, we need to use some commands to configure the network, but in the network configuration, we must understand some knowledge of virtual machines, so the previous knowledge of virtual machines needs to be more solid.

Note here that CentOS7 and CentOS6 commands are somewhat different, some common commands are necessary, each boot or start decompression installation authorization is the most basic knowledge, others can not be learned for the time being.

Install java environment in CentOS

Download the corresponding 32-bit or 64-bit jdk, set the JDK environment variable PATH, and you can Google the specific configuration.

Environmental preparation

Change the machine name, like this in every system.

Open the terminal and switch to the root user to modify the machine name.

$sudo su

$vim / etc/sysconfig/network

Add "HOSTNAME=H32" to it

Add the machine name corresponding to each server IP

All three ip addresses are added to their respective / etc/hosts files

Every system has to be modified like this.

# Editing hosts

Vim / etc/hosts

Close SELinux

# Editing SELinux configuration file

Vim / etc/selinux/config

# change status

SELINUX=disabled

SSH Settin

Enter H30 to see if ssh is installed, and if so, continue without installation.

Rpm-qa | grep ssh

Time server installation

Time servers need to be installed here, and other servers use this to synchronize multi-machine time. Inconsistent machine time in distributed deployment will lead to data errors or intra-component communication errors.

Install the httpd service

Httpd is the main program of the Apache Hypertext transfer Protocol (HTTP) server. Designed as a background process that runs independently, it establishes a pool of child processes or threads that process requests.

Install yum-utils

Yum install yum-utils for online yum installation it is based on RPM package management, can automatically download RPM packages from the designated server and install, can automatically deal with dependencies, and install all dependent software packages at a time, without tedious downloading and installation. Yum provides commands to find, install, and delete one, a group, or even all of the software packages, and the commands are concise and easy to remember. (requires the machine to be able to access the external network)

Prepare resources and environment

Download the iso installation image of Centos 7.2, jdk1.8 for Linux package, hadoop2.7.3 package (originally using hadoop3, but found to be not compatible with the latest version of hive)

Basic knowledge of Java

Since the source code of the Hadoop framework is written in Java, it is best to use Java as the best language for development, although some of them have been encapsulated in C #, but their performance has been greatly reduced. (this is why learning from big data in Java has a great advantage, while those in other languages need to relearn Java.)

Eclipse IDE usage

Source Code Compiler Technology under Linux and Windows

And the most important thing is that your Internet speed should be good, because the dependency package will be downloaded during the installation process, and the bad network will make you doubt your life.

Finally, you need to be patient. The environment of each machine is different, and the version of the package downloaded by each person may lead to all kinds of problems. At this time, you need to be patient to read the log, keep trying and carefully compare the installation steps. It may be a missing variable or a case error or a restart.

In order to help you make learning easier and efficient, we will share a large number of materials free of charge to help you overcome difficulties on your way to becoming big data engineers and even architects. Here to recommend a big data learning exchange circle: 658558542 welcome everyone to enter × × × stream discussion, learning exchange, common progress.

When you really start learning, it is inevitable that you do not know where to start, resulting in inefficiency that affects your confidence in continuing learning.

But the most important thing is not to know which skills need to be mastered, step on the pit frequently while learning, and eventually waste a lot of time, so it is necessary to have effective resources.

Finally, I wish all the big data programmers who encounter bottle disease and do not know what to do, and wish you all every success in the future work and interview.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.