Shulou(Shulou.com)06/03 Report--

Hadoop

Install the Ubuntu environment

192.168.1.64 HNClient

192.168.1.65 HNName

SUSE,Ubuntu vi cannot use the Backspace key to delete data

When deleting, press ESC and then X to delete the data

Insert data, using I

Open a new line under the current line, using o

Operate on HNClient

Norman@HNClient:~$ sudo vi / etc/hostname

Norman@HNClient:~$ HNClient

Norman@HNClient:~$ sudo apt-get install openssh-server

Norman@HNClient:~$ sudo vi / etc/hosts

192.168.1.64 HNClient

192.168.1.65 HNName

Norman@HNClient:~$ ssh-keygen (enter by default below)

Generating public/private rsa key pair.

Enter file in which to save the key (/ home/norman/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in / home/norman/.ssh/id_rsa.

Your public key has been saved in / home/norman/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:rj3kM5OeqxceqGP6DcofXa+hZFReLQmKqksqoYL+YH4 norman@HNClient

The key's randomart image is:

+-[RSA 2048]-+

|. | |

|. . . O |

|. . . +. | |

|. O. . | |

|. .. s |

|. O.o.oz. | |

| | + = o.roommates. | | |

| | Xo.E+ + X+ | |

| | oo.=+ |

+-[SHA256]-+

Norman@HNClient:~$ ssh localhost (ssh localhost, still requires password authentication)

Norman@localhost's password:

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

251 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 23:14:08 2018 from 192.168.1.65

Norman@HNClient:~$ cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys

Norman@HNClient:~$ ssh localhost (ssh localhost, no password authentication required)

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

251 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 23:18:02 2018 from 127.0.0.1

Norman@HNClient:~$ ssh HNName (ssh HNName, still requires password authentication)

Norman@hnname's password:

Norman@HNClient:~$ ssh-copy-id-I / .ssh/id_rsa.pub norman@HNName

Norman@HNClient:~$ ssh HNName (ssh HNName, you can log in to HNName without a password)

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

254 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 23:23:21 2018 from 192.168.1.64

Norman@HNName:~$

Operate on HNName

Norman@HNName:~$ sudo vi / etc/hosts

192.168.1.64 HNClient

192.168.1.65 HNName

Norman@HNName:~$ ssh-keygen (enter by default below)

Generating public/private rsa key pair.

Enter file in which to save the key (/ home/norman/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in / home/norman/.ssh/id_rsa.

Your public key has been saved in / home/norman/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:YXrPGdhKYkPsAroDlIZJ4sYdbrpHyvaMQccMV3GJn9I norman@HNName

The key's randomart image is:

+-[RSA 2048]-+

|. . Oo.. | |

|. + oo.. | |

| | oO.= = + | |

| | + .B. + E + |

| oo =. B S o |

| | +. = o = + o |

| o. . + |

|. * |

|. O |

+-[SHA256]-+

Norman@HNClient:~$ ssh localhost (ssh localhost, still requires password authentication)

Norman@localhost's password:

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

251 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 22:55:29 2018 from 127.0.0.1

Norman@HNName:~$ cat ~ / .ssh/id_rsa.pub > > ~ / .ssh/authorized_keys

Norman@HNName:~$ ssh localhost (ssh localhost, no password authentication required)

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

254 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 23:00:28 2018 from 127.0.0.1

Norman@HNName:~$ ssh-copy-id-I / .ssh/id_rsa.pub norman@hnclient

/ usr/bin/ssh-copy-id: INFO: Source of key (s) to be installed: "/ home/norman/.ssh/id_rsa.pub"

The authenticity of host 'hnclient (192.168.1.64)' can't be established.

ECDSA key fingerprint is SHA256:w5dwBrXor00JfFtpGXc0G/+deJJwmAxKmjXE32InhgA.

Are you sure you want to continue connecting (yes/no)? Yes

/ usr/bin/ssh-copy-id: INFO: attempting to log in with the new key (s), to filter out any that are already installed

/ usr/bin/ssh-copy-id: INFO: 1 key (s) remain to be installed-- if you are prompted now it is to install the new keys

Norman@hnclient's password:

Number of key (s) added: 1

Now try logging into the machine, with: "ssh 'norman@hnclient'"

And check to make sure that only the key (s) you wanted were added.

Norman@HNName:~$ ssh hnclient (ssh hnclient, you can log in to hnclient without a password)

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.13.0-36-generic i686)

Documentation: https://help.ubuntu.comManagement: https://landscape.canonical.comSupport: https://ubuntu.com/advantage

251 packages can be updated.

79 updates are security updates.

New release '18.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Wed Oct 31 23:05:13 2018 from 192.168.1.58

Norman@HNClient:~$ exit

Norman@HNName:~$ sudo apt-get install openjdk-7-jdk

[sudo] password for norman:

Reading package lists... Done

Building dependency tree

Reading state information... Done

Package openjdk-7-jdk is not available, but is referred to by another package.

This may mean that the package is missing, has been obsoleted, or

Is only available from another source

E: Package 'openjdk-7-jdk' has no installation candidate

It is because the installation source of Ubuntu16.04 does not have openjdk7 by default, so you have to manually add the repository yourself, as follows:

Norman@HNName:~$ sudo add-apt-repository ppa:openjdk-r/ppa (add oracle openjdk ppa source) (add-apt-repository ppa: xxx/ppa) means to get the latest personal package archive source, add it to the current apt library, and automatically import the public key.)

Norman@HNName:~$ sudo apt-get update

Norman@HNName:~$ sudo apt-get install openjdk-7-jdk

Norman@HNName:~$ java-version

Java version "1.7.0Y95"

OpenJDK Runtime Environment (IcedTea 2.6.4) (7u95-2.6.4-3)

OpenJDK Client VM (build 24.95-b01, mixed mode, sharing)

Norman@HNName:~$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.0/hadoop-1.2.0-bin.tar.gz

Norman@HNName:~$ tar-zxvf hadoop-1.2.0-bin.tar.gz

Norman@HNName:~$ sudo cp-r hadoop-1.2.0 / usr/local/hadoop

Norman@HNName:~$ dir / usr/local/hadoop

Bin hadoop-ant-1.2.0.jar hadoop-tools-1.2.0.jar NOTICE.txt

Build.xml hadoop-client-1.2.0.jar ivy README.txt

C++ hadoop-core-1.2.0.jar ivy.xml sbin

CHANGES.txt hadoop-examples-1.2.0.jar lib share

Conf hadoop-minicluster-1.2.0.jar libexec src

Contrib hadoop-test-1.2.0.jar LICENSE.txt webapps

Norman@HNName:~$ sudo vi $HOME/.bashrc (add the following at the end)

Export HADOOP_PREFIX=/usr/local/hadoop

Export PATH=$PATH:$HADOOP_PREFIX/bin

Norman@HNName:~$ exec bash

Norman@HNName:~$ $PATH

Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/hadoop-env.sh

(The java implementation to use. Required.)

Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-i386

(Extra Java runtime options. Empty by default. Set to disable IPv6)

Export HADOOP_OPTS=-Djava.net.preferIP4Stack=true

Installing Apache Hadoop (Single Node)

Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/core-site.xml

Fs.default.name

Hdfs://HNName:10001

Hadoop.tmp.dir

/ usr/local/hadoop/tmp

Norman@HNName:~$ sudo vi / usr/local/hadoop/conf/mapred-site.xml

Mapred.job.tracker

HNName:10002

Norman@HNName:~$ sudo mkdir / usr/local/hadoop/tmp

Norman@HNName:~$ sudo chown norman / usr/local/hadoop/tmp

Norman@HNName:~$ hadoop namenode-format (you can see the following description of success)

18-11-01 19:07:36 INFO common.Storage: Storage directory / usr/local/hadoop/tmp/dfs/name has been successfully formatted.

Norman@HNName:~$ hadoop-daemons.sh start namenode (with the following error)

Localhost: mkdir: cannot create directory? usr/local/hadoop/libexec/../logs? Permission denied

Localhost: chown: cannot access'/ usr/local/hadoop/libexec/../logs': No such file or directory

Localhost: starting namenode, logging to / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out

Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 137: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory

Localhost: head: cannot open'/ usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out' for reading: No such file or directory

Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 147: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory

Localhost: / usr/local/hadoop/bin/hadoop-daemon.sh: line 148: / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out: No such file or directory

Norman@HNName:~$ ll / usr/local

Total 44

Drwxr-xr-x 11 root root 4096 Nov 1 02:02. /

Drwxr-xr-x 11 root root 4096 Feb 28 2018.. /

Drwxr-xr-x 2 root root 4096 Feb 28 2018 bin/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 etc/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 games/

Drwxr-xr-x 15 root root 4096 Nov 1 20:05 hadoop/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 include/

Drwxr-xr-x 4 root root 4096 Feb 28 2018 lib/

Lrwxrwxrwx 1 root root 9 Jul 26 23:29 man-> share/man/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 sbin/

Drwxr-xr-x 8 root root 4096 Feb 28 2018 share/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 src/

Norman@HNName:~$ sudo chown norman / usr/local/hadoop

Norman@HNName:~$ ll / usr/local

Total 44

Drwxr-xr-x 11 root root 4096 Nov 1 02:02. /

Drwxr-xr-x 11 root root 4096 Feb 28 2018.. /

Drwxr-xr-x 2 root root 4096 Feb 28 2018 bin/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 etc/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 games/

Drwxr-xr-x 15 norman root 4096 Nov 1 20:05 hadoop/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 include/

Drwxr-xr-x 4 root root 4096 Feb 28 2018 lib/

Lrwxrwxrwx 1 root root 9 Jul 26 23:29 man-> share/man/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 sbin/

Drwxr-xr-x 8 root root 4096 Feb 28 2018 share/

Drwxr-xr-x 2 root root 4096 Feb 28 2018 src/

Norman@HNName:~$ hadoop-daemons.sh start namenode

Localhost: starting namenode, logging to / usr/local/hadoop/libexec/../logs/hadoop-norman-namenode-HNName.out

Norman@HNName:~$ start-all.sh

Norman@HNName:~$ jps

23297 DataNode

23610 TaskTracker

23484 JobTracker

23739 Jps

23102 NameNode

23416 SecondaryNameNode

Norman@HNName:~$ dir / usr/local/hadoop/bin

Hadoop hadoop-daemon.sh rcc start-all.sh start-dfs.sh start-mapred.sh stop-balancer.sh stop-jobhistoryserver.sh task-controller

Hadoop-config.sh hadoop-daemons.sh slaves.sh start-balancer.sh start-jobhistoryserver.sh stop-all.sh stop-dfs.sh stop-mapred.sh

Http://192.168.1.65:50070/dfshealth.jsp

Http://192.168.1.65:50030/jobtracker.jsp

Http://192.168.1.65:50060/tasktracker.jsp

Managing HDFS

Http://www.gutenberg.org/files/2600/2600-0.txt (download text file)

Copy web page content to war_and_peace.txt

Https://www.ncdc.noaa.gov/orders/qclcd/ (download any data)

QCLCD201701.zip,QCLCD201702.zip, and then extract 201701hourly.txt, 201702hourly.txt

Operate on HNClient

Put the data war_and_peace.txt to / home/norman/data/book

Put the data 201701hourly.txtjol 201702hourly.txt to / home/norman/data/weather

Norman@HNClient:~$ sudo mkdir-p / home/norman/data/book

Norman@HNClient:~$ sudo mkdir-p / home/norman/data/weather

Norman@HNClient:~$ sudo chown norman/ home/norman/data/weather

Norman@HNClient:~$ sudo chown norman/ home/norman/data/book

Norman@HNClient:~$ sudo add-apt-repository ppa:openjdk-r/ppa

Norman@HNClient:~$ sudo apt-get update

Norman@HNClient:~$ sudo apt-get install openjdk-7-jdk

Norman@HNClient:~$ java-version

Norman@HNClient:~$ wget http://archive.apache.org/dist/hadoop/core/hadoop-1.2.0/hadoop-1.2.0-bin.tar.gz

Norman@HNClient:~$ tar-zxvf hadoop-1.2.0-bin.tar.gz

Norman@HNClient:~$ sudo cp-r hadoop-1.2.0 / usr/local/hadoop

Norman@HNClient:~$ sudo vi $HOME/.bashrc (add the following at the end)

Export HADOOP_PREFIX=/usr/local/hadoop

Export PATH=$PATH:$HADOOP_PREFIX/bin

Norman@HNClient:~$ exec bash

Norman@HNClient:~$ $PATH

Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/hadoop-env.sh

(The java implementation to use. Required.)

Export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-i386

(Extra Java runtime options. Empty by default. Set to disable IPv6)

Export HADOOP_OPTS=-Djava.net.preferIP4Stack=true

Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/core-site.xml

Fs.default.name

Hdfs://HNName:10001

Hadoop.tmp.dir

/ usr/local/hadoop/tmp

Norman@HNClient:~$ sudo vi / usr/local/hadoop/conf/mapred-site.xml

Mapred.job.tracker

HNName:10002

Norman@HNClient:~$ hadoop fs-mkdir test

Norman@HNClient:~$ hadoop fs-ls

Found 1 items

Drwxr-xr-x-norman supergroup 0 2018-11-02 01:17 / user/norman/test

Norman@HNClient:~$ hadoop fs-mkdir hdfs://hnname:10001/data/small

Norman@HNClient:~$ hadoop fs-mkdir hdfs://hnname:10001/data/big

Open the web page to http://192.168.1.65:50070

Http://192.168.1.65:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/

Norman@HNClient:~$ hadoop fs-rmr test (Test deletion)

Deleted hdfs://HNName:10001/user/norman/test

Norman@HNClient:~$ hadoop fs-moveFromLocal / home/norman/data/book/war_and_peace.txt hdfs://hnname:10001/data/small/war_and_peace.txt

You can see the following data

Norman@HNClient:~$ hadoop fs-copyToLocal hdfs://hnname:10001/data/small/war_and_peace.txt / home/norman/data/book/war_and_peace.bak.txt (test replicated locally)

Norman@HNClient:~$ hadoop fs-put / home/norman/data/weather hdfs://hnname:10001/data/big

You can see the following data

Norman@HNClient:~$ hadoop dfsadmin-report

Configured Capacity: 19033165824 (17.73 GB)

Present Capacity: 13114503168 (12.21 GB)

DFS Remaining: 12005150720 (11.18 GB)

DFS Used: 1109352448 (1.03GB)

DFS Used%: 8.46%

Under replicated blocks: 19

Blocks with corrupt replicas: 0

Missing blocks: 0

Datanodes available: 1 (1 total, 0 dead)

Name: 192.168.1.65:50010

Decommission Status: Normal

Configured Capacity: 19033165824 (17.73 GB)

DFS Used: 1109352448 (1.03GB)

Non DFS Used: 5918662656 (5.51 GB)

DFS Remaining: 12005150720 (11.18 GB)

DFS Used%: 5.83%

DFS Remaining%: 63.07%

Last contact: Fri Nov 02 01:49:43 GMT-08:00 2018

Norman@HNClient:~$ hadoop dfsadmin-safemode enter (safemode is required for upgrade)

Safe mode is ON

Norman@HNClient:~$ hadoop dfsadmin-safemode leave

Safe mode is OFF

Operate on HNName

Norman@HNName:~$ hadoop fsck-blocks

Status: HEALTHY

Total size: 1100586452 B

Total dirs: 13

Total files: 4

Total blocks (validated): 19 (avg. Block size 57925602 B)

Minimally replicated blocks: 19 (100.0)

Over-replicated blocks: 0 (0.0%)

Under-replicated blocks: 19 (100.0)

Mis-replicated blocks: 0 (0.0%)

Default replication factor: 3

Average block replication: 1.0

Corrupt blocks: 0

Missing replicas: 38 (200.0)

Number of data-nodes: 1

Number of racks: 1

FSCK ended at Fri Nov 02 01:54:46 GMT-08:00 2018 in 1049 milliseconds

The filesystem under path'/'is HEALTHY

Norman@HNName:~$ hadoop fsck / data/big

Status: HEALTHY

Total size: 1097339705 B

Total dirs: 2

Total files: 2

Total blocks (validated): 17 (avg. Block size 64549394 B)

Minimally replicated blocks: 17 (100.0)

Over-replicated blocks: 0 (0.0%)

Under-replicated blocks: 17 (100.0)

Mis-replicated blocks: 0 (0.0%)

Default replication factor: 3

Average block replication: 1.0

Corrupt blocks: 0

Missing replicas: 34 (200.0)

Number of data-nodes: 1

Number of racks: 1

FSCK ended at Fri Nov 02 19:33:55 GMT-08:00 2018 in 14 milliseconds

The filesystem under path'/ data/big' is HEALTHY

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.