Shulou(Shulou.com)06/03 Report--

I built it according to the following article:

Http://cio.51cto.com/art/201705/539356.htm

To be honest, the above article is very good. It hardly needs to be changed. Build it step by step, no problem.

The pit I encountered is:

Master cannot connect to slave. What to do. The configuration files are fine.

Solution: namenode has been formatted many times, resulting in multiple clusterID. (I don't dare to format and play in the future.)

Just make sure that the clusterID and slaveID of master are the same, and you can connect.

My path is: master: hadoop273/hdfs/name/current/VERSION

Slave: hadoop273/hdfs/data/data/VERSION

Just like the clusterID in VERSION. Nothing else needs to be changed. (there is no need to change the layoutVersion.

The end of this article is also useful: http://blog.csdn.net/zzu09huixu/article/details/24725769

And this one: http://blog.csdn.net/longzilong216/article/details/20648387

= =

Hostname problem: the hostname should be the same in / etc/hosts and / etc/sysconfig/network.

I have changed these two files, but I still don't recognize the host name. The ping hostnames are all ping impassable. I dare not restart it. So

Hostname "hostname", which takes effect temporarily with a command. It got through.

= =

Java, I installed it myself. Articles found on the Internet:

Http://www.cnblogs.com/shihaiming/p/5809553.html

After installation, it will be OK.

Remember, the java version, and the version in the article with hadoop, should be matched accordingly. I mean, the name should be changed, for example, the article I pretended to be jdk1.8.0_131.hadoop was jdk1.8.0_101. Just change the name of the folder.

4. Please =

Don't change the configuration in this article.

Hadoop just started, and I installed it with root. Later, in order to use hadoop as a user. I changed the group and the owner. And hadoop273 (folder) and java (folder) permissions, all 777.

Chown-R hadoop:hadoop hadoop273

Chmod-R 777 hadoop273

Now, hadoop can be used.

Later, for the sake of other users' rights assignment problems. I changed the folder permissions again (back up before changing):

Chmod-R 755 hadoop273

Chmod-R 755 java

5. Test commands:

What I use is (copied from others):

Run the sample program:

Create a folder on hdfs first

. / bin/hdfs dfs-mkdir / input

. / bin/hadoop jar. / share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarrandomwriter input

It is the above two commands, ok pull. Randomwriter, I don't know whether this is a file or a folder. It's not on my system anyway, but if you hit these two commands, you can test it. It's true.

6.ssh secret-free login:

At startup:

Sbin/start-dfs.sh

Sbin/start-yarn.sh

Or: sbin/start-all.sh

Just these commands, you have to enter the password every time. And enter it 4 times. It's annoying. Even copying and pasting is still annoying.

I solved my problem with the following connection:

Http://blog.csdn.net/firehadoop/article/details/68953541

Just copy the public key of master to slave. There is no need to enter a password for master to slave.

The other way around. Slave to master, and you have to enter the password.

At first, I thought that if you start the cluster on master, you have to start it again on slave. It turns out that as long as it is started on master, it will start automatically on slave. Oh, I just got it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.