Shulou(Shulou.com)06/03 Report--

The separate\ in the script is to replace one line with multiple lines, newline characters.

The wildcard 0.0.0.0 means that any ip can be accessed, the computer has a network card, and the other ones are connected locally.

2NN configures a separate host-it is now 005 in 18 minutes and 38 seconds

-

Dfs.namenode.secondary.http-address

0.0.0.0 DFSUtil 50090-here, the above s500 50090 extracts all configuration items from the DFSUtil package-search the configuration found in the secondary in the little red book and the configuration found in the source code in the Redbook

The secondary namenode http server address and port.

Change the secondary name of the node / etc/hadoop_cluster/hdfs-site.xml to [add] the following content in this file-why this is it, because what is found above is the content of hdfs-default

Dfs.namenode.secondary.http-address

S500:50090

Send this file to all nodes after configuration

Hdfs-- config / soft/hadoop/etc/hadoop_cluster getconf-secondarynamenodes

Take it out and it's S500.

Configure the new cluster and reformat it

Modify the default hadoop temporary directory-if you want to modify it, you can modify it here. If you want to search the configuration, search local in the little red book and search hadoop.tmp.dir.

-these contents are appended, and after the append is sent to other hosts-I am a ubuntu user, so I will put it in my own home. If not, create a directory and create a directory on each machine.

[core-site.xml]

Hadoop.tmp.dir=/home/ubuntu/hadoop/

=

It is best to delete the log in soft/hadoop/logs before cloning.

Configure a fully distributed xxx-site.xml

S100 namenode name Node

S200 datanode data node

S300 datanode data Node

S400 datanode data node

The S500 secondarynamenode auxiliary name node mentioned that slaves is not easy to use in 17 minutes and 44 seconds at 003. Let's see the source code.

Slave matching

S200

S300

S400

Etc/haoop_cluster is sent to another host

Hdfs-config / soft/hadoop/etc/hadoop_cluster namenode-format formatting also mentions the relevant configuration

After formatting, you can read it in / tmp/hadoop-ubuntu/dfs/.

Start-dfs.sh-config / soft/hadoop/etc/hadoop_cluster

Start-yarn.sh-config / soft/hadoop/etc/hadoop_cluster

It is also completely distributed related operations.

Hadoop port

-

1.namenode 50070

Http://namenode:50070/

2.resourcemanager:8088

Http://localhost:8088/

3.historyServer

Http://hs:19888/

4.name rpc (remote procedure call, remote procedure call)-the first

Hdfs://namenode:8020/ host name: Port number

Ssh instruction combined with operation command

-

$> ssh s300 rm-rf / xx/x/x forcibly delete directories and kill them recursively

Ssh s300 rm-rf a b delete a directory, b directory

Remote replication via scp

-

Recursive copy of $> scp-r / xxx/x ubuntu@s200:/path directory

Scp-r hadoop_cluster ubuntu@s200:/soft/hadoop/etc/

Write scripts to copy files or folders remotely on all nodes.

Xcopy.sh-remember to modify the permissions that everyone can execute-there is this directory in path[ / usr/local/sbin:/usr/local/bin] under / usr/local/sbin, and we don't have to accompany.

-

1.

two。

3.

4.

Scp-r path ubuntu@s200:/path

Delete

-

Xrm.sh a.txt

Ssh S200 rm-rf path

-- delete script-003 start-what we often delete are / soft/hadoop/logs directory and / tmp directory. This tmp directory will not be generated without formatting.

#! / bin/bash

If [$#-lt 1]; then

Echo no args

Exit

Fi

# get first argument

Arg1=$1

Cuser= `whoami`

Fname= `basename $arg1`

Dir= `dirname $arg1`

If ["$dir" = "."]; then

Dir= `pwd`

Fi

For ((iTune100bomachi 8m), check the block size through webui

Hadoop--config / soft/hadoop/etc/hadoop_cluster fs-put / mnt/hgfs/downloads/bigdata/hadoop-2.7.2.tar.gz / usr/ubuntu/data/

Other hosts such as s200 and S300 can also put files.

Hadoop-- config / soft/hadoop/etc/hadoop_cluster fs-chmod change permission

Check the status of the file block hadoop fsck /-blocks

The teacher called hdfs-- config / soft/hadoop/etc/hadoop_cluster fsck / user/ubuntu/data/hello.txt to check the health status of the documents.

Go to the temporary directory and cat the meta and block. Each node has the same content.

Go to webui to see how many files with more than 128pieces

[ip] the mapping configuration in hosts depends on the ip of your host.

S100-> 132

S200-> 130

S300-> 131

S400-> 128

S500-- > 133

127.0.0.1localhost

192.168.238.132 s100

192.168.238.130 s200

192.168.238.131 s300

192.168.238.128 s400

192.168.238.133 s500

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.