Shulou(Shulou.com)06/03 Report--

Github of Fayson: https://github.com/fayson/cdhproject

It is recommended to follow the official account of Wechat: "Hadoop practice", ID:gh_c4c535955d0f, or scan the QR code at the end of the article.

1. Purpose of document writing

The Gateway node, also known as the client node, is usually used as an interface machine to access the Hadoop cluster. It will mainly deploy some client configuration, script commands, such as HDFS core-site.xml,hdfs-site.xml and hadoop operation commands.

If you are using Apache Hadoop, you only need to copy the configuration and scripting commands of the hadoop-related services to the client machine, but once the configuration of the cluster is changed, you need to synchronize to the client machine as well. If it is a CDH cluster, the client node will also be a machine managed by Cloudera Manager, and it will be installed with cloudera-scm-agent service and Parcel of CDH. The deployment client configuration Cloudera Manager will be done uniformly. In addition, if there is an exception on the client machine, Cloudera Manager will also alarm.

Adding a Gateway node is very similar to installing CDH, you must be careful to prepare the client machine, refer to "CDH installation preparation", otherwise it will increase failures. Earlier, Fayson described deploying Gateway nodes in a non-Kerberos environment. Refer to "how to add Gateway nodes to a CDH cluster". This article mainly introduces how to add Gateway nodes to the CDH cluster in the Kerberos environment.

Content Overview

1. Create a host template for the Gateway node

Pre-preparation of 2.Gateway nodes

3. Add Gateway nodes to the cluster and apply host templates

4.GateWay Node Command Test

Test environment

1.CDH5.13

two。 Adopt root user operation

3.CentOS6.5

Precondition

The 1.CDH5.13 cluster is running normally

The 2.Gateway node is ready and ready to be prepositioned

two。 Create a host template for the Gateway node

1. Go to the Host template page from Cloudera Manager

two。 Click "create"

3. Name the template, click on each service and check the corresponding GateWay role.

4. Click create to confirm the success of the creation.

Pre-preparation of 3.Gateway nodes

For pre-preparation, please refer to the previous article "CDH installation pre-preparation" of Fayson, which mainly includes the following steps:

1. Make sure that the yum source of OS can be used properly, and all packages of matching OS can be viewed through the yum repolist command.

two。 Make sure the yum source for Cloudera Manager is running properly

3.hosts file configuration, the IP and hostname of the Gateway node need to be added to the hosts file of the CDH cluster node, and synchronized to all machines, including the Gateway node

4. Disable SELinux

5. Turn off the firewall

6. Set swap to 10

7. Close large transparent pages

8. Configure clock synchronization

Please make sure that the above operations have been completed and configured successfully, otherwise the next additional node operations will fail!

4. Install the Kerberos client

Because the Kerberos service is enabled in the cluster, the Kerberos client needs to be installed on the Gateway node.

1. Execute the following command on the Gateway node

[ec2-user@ip-172-31-31-212opt] $sudo yum-y install krb5-libskrb5-workstation

Check the installed RPM package after successful installation

[ec2-user@ip-172-31-31-212opt] $rpm-qa | grep krbkrb5-workstation-1.15.1-8.el7.x86_64krb5-libs-1.15.1-8.el7.x86_64krb5-devel-1.15.1-8.el7.x86_64 [ec2-user@ip-172-31-31-212opt] $

two。 Copy the krb5.conf file from the CM cluster to the Gateway node

[ec2-user@ip-172-31-22-86] $scp-I fayson.pem.txt / etc/krb5.conf ip-172-31-31-212:/home/ec2-user/

3. Copy the krb5.conf file to the / etc directory in the Gateway node

[ec2-user@ip-172-31-31-212] $sudo cp krb5.conf / etc/ [ec2-user@ip-172-31-31-212] $sudo chown root. / etc/krb5.conf [ec2-user@ip-172-31-31-212 ~] $ll / etc/krb5.conf-rw-r--r-- 1 root root 837 Dec 9 00:28 / etc/krb5.conf [ec2-user@ip-172-31-31-212 ~] $

4. Test whether the Kerberos client is deployed successfully on the GateWay node

[ec2-user@ip-172-31-31-21212] $kinit-kt fayson.keytab fayson [ec2-user@ip-172-31-31-121212] $klistTicket cache: FILE:/tmp/krb5cc_1000Default principal: fayson@CLOUDERA.COMValid starting Expires Service principal12/09/2017 00:31:53 12 krbtgt/CLOUDERA.COM@CLOUDERA.COM renew until 10, 2017 00:31:53 krbtgt/CLOUDERA.COM@CLOUDERA.COM renew until 12 pound 16 2017 00:31:53 [ec2-user@ip-172-31-31-212] $

As shown in the figure above, the Kerberos client is installed successfully.

5. Add clusters of Gateway nodes and apply host templates

1. Go to the all hosts page

two。 Click "add hosts to Cluster"

3. Select Classic Wizard

4. Go ahead

5. Enter the IP or hostname of the Gateway node, and click search

6. Click continue, select Custom Repository, and enter the yum source http address of Cloudera Manager

7. Click "continue" and check the two options for Java

8. Click continue and enter the ec2-user password of the Gateway node

9. Click continue and wait for cloudera-scm-agent to be installed on the Gateway node

Click "continue" when the installation is complete

10. Click "continue" and wait for the Parcel package to be distributed and activated

When you are finished, click continue

11. Click "continue" to check the host

twelve。 Complete the host check, click "continue" and select the host template

13. Click continue to start the role on the host

Waiting for the execution to succeed

14. Click "continue" to deploy the client configuration

15. Click finish to view the role information of the GateWay node in the host list

At this point, a new Gateway node is added to the CDH cluster in the Kerberos environment.

6.Gateway node test

1.HDFS command test

[ec2-user@ip-172-31-31-212] $hadoop fs-ls /

2.HBase command test

[ec2-user@ip-172-31-31-212] $hbase shell

3.Hive command test

[ec2-user@ip-172-31-31-212] $hive

4.hadoop command to submit jobs to the cluster

[ec2-user@ip-172-31-31-212] $hadoop jar / opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 55

Set your mind for heaven and earth, set your life for the people, continue to learn for the past, and open peace for all eternity.

It is recommended to follow Hadoop practice, the first time, share more Hadoop practical information, welcome to forward and share.

Original article, welcome to reprint, reprint please indicate: reproduced from the official account of Wechat Hadoop

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.