Shulou(Shulou.com)06/02 Report--

This article mainly introduces "how to enable Kerberos in the CDH cluster". In the daily operation, I believe many people have doubts about how to enable Kerberos in the CDH cluster. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts of "how to enable Kerberos in the CDH cluster"! Next, please follow the editor to study!

1. Purpose of document writing

This document describes how to enable and configure Kerberos in a CDH cluster. You will learn the following:

1. How to install and configure KDC services

two。 How to enable Kerberos through CDH

3. How to log in to Kerberos and access Hadoop related services

The document is mainly divided into the following steps:

1. Install and configure KDC services

2.CDH cluster enables Kerberos

3.Kerberos usage

This document focuses on how to enable and configure Kerberos in a CDH cluster, based on the following assumptions:

The 1.CDH cluster is running normally

two。 Kerberos is not enabled in the cluster

3.MySQL 5.1.73

The following is the test environment, but not required for this operation manual:

1. Operating system: CentOS 6.5

Version 5.12.0 for 2.CDH and CM

3. Use root users to operate

2.KDC service installation and configuration

In this document, KDC services are installed on the same server as Cloudera Manager Server (KDC services can be installed on other servers according to their own needs)

1. Install the KDC service on the Cloudera Manager server

[root@ip-172-31-6-148C] # yum-y install krb5-serverkrb5-libs krb5-auth-dialog krb5-workstation

two。 Modify / etc/krb5.conf configuration

[root@ip-172-31-6-148fayson_r] # vim / etc/krb5.conf [logging] default = FILE:/var/log/krb5libs.log kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log [libdefaults] default_realm = FAYSON.COM dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24 h renew_lifetime = 7d forwardable = true [realms] FAYSON.COM = {kdc = ip-172-31-6 -148.fayson.com admin_server = ip-172-31-6-148.fayson.com} [domain_realm] .IP-172-31-6-148.fayson.com = FAYSON.COM ip-172-31-6-148.fayson.com = FAYSON.COM

The red section is the information that needs to be modified.

3. Modify / var/kerberos/krb5kdc/kadm5.acl configuration

[root@ip-172-31-6-148] # vim / var/kerberos/krb5kdc/kadm5.acl * / admin@FAYSON.COM *

4. Modify / var/kerberos/krb5kdc/kdc.conf configuration

# vim / var/kerberos/krb5kdc/kdc.conf [kdcdefaults] kdc_ports = 88 kdc_tcp_ports = 88 [realms] FAYSON.COM= {# master_key_type = aes256-cts max_renewable_life= 7d 0h 000s acl_file = / var/kerberos/krb5kdc/kadm5.acl dict_file = / usr/share/dict/words admin_keytab = / var/kerberos/krb5kdc/kadm5.keytab supported_enctypes = aes256-cts:normal aes128-cts : normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal}

The red section is the configuration that needs to be modified.

5. Create a Kerberos database

[root@ip-172-31-6-148C] # kdb5_util create-r FAYSON.COM-sLoading random dataInitializing database'/ var/kerberos/krb5kdc/principal' for realm 'FAYSON.COM',master key name' K/M@FAYSON.COM'You will be prompted for the database Master Password.It is important that you NOT FORGET this password.Enter KDC database master key: Re-enter KDC database master key to verify:

You need to enter the password for the Kerberos database here.

6. Create an administrative account for Kerberos

# kadmin.localAuthenticating as principal fayson/admin@CLOUDERA.COM with password.kadmin.local: addprinc admin/admin@FAYSON.COMWARNING: no policy specified for admin/admin@FAYSON.COM; defaulting to no policyEnter password for principal "admin/admin@FAYSON.COM": Re-enter password for principal "admin/admin@FAYSON.COM": Principal "admin/admin@FAYSON.COM" created.kadmin.local: exit [root@ip-172-31-6-148C] #

The red part is the Kerberos administrator account, and you need to enter the administrator password.

7. Add the Kerberos service to the self-startup service and start the krb5kdc and kadmin services

[root@ip-172-31-6-148~] # chkconfig kadmin on [root @ ip-172-31-6-148~] # service krb5kdc startStarting Kerberos 5 KDC: [OK] [root@ip-172-31-6-148~] # service kadmin startStarting Kerberos 5 Admin Server: [OK ] [root@ip-172-31-6-14814] #

8. Test the administrator account of Kerberos

[root@ip-172-31-6-14814] # kinit admin/admin@FAYSON.COMPassword for admin/admin@FAYSON.COM: [root@ip-172-31-6-14814] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: admin/admin@FAYSON.COMValid starting Expires Service principal09/05/17 16:39:17 09 krbtgt/FAYSON.COM@FAYSON.COM renew until 06 kinit admin/admin@FAYSON.COMPassword for admin/admin@FAYSON.COM 17 16:39:17 krbtgt/FAYSON.COM@FAYSON.COM renew until 09 Grey 17 16:39 17 [root@ip-172-31-6-14814] #

9. Install all Kerberos clients, including Cloudera Manager, for the cluster

[root@ip-172-31-6-148cdh-shell-master] # yum-y install krb5-libs krb5-workstation

10. Install additional packages on the Cloudera Manager Server server

[root@ip-172-31-6-148cdh-shell-master] # yum-y install openldap-clients

11. Copy the krb5.conf file on KDC Server to all Kerberos clients

[root@ip-172-31-6-148cdh-shell-master] # scp-r / etc/krb5.conf root@172.31.5.190:/etc/

Use scripts to copy here

[root@ip-172-31-6-148cdh-shell-master] # sh b.sh node.list / etc/krb5.conf / etc/krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 krb5.conf 100% 451 0.4KB/s 00:00 [root@ip-172-31-6-148cdh-shell-master] #

3.CDH cluster enables Kerberos

1. Add administrator account to Cloudera Manager in KDC

[root@ip-172-31-6-148cdh-shell-bak] # kadmin.localAuthenticating as principal admin/admin@FAYSON.COM with password.kadmin.local: addprinc cloudera-scm/admin@FAYSON.COMWARNING: no policy specified for cloudera-scm/admin@FAYSON.COM Defaulting to no policyEnter password for principal "cloudera-scm/admin@FAYSON.COM": Re-enter password for principal "cloudera-scm/admin@FAYSON.COM": Principal "cloudera-scm/admin@FAYSON.COM" created.kadmin.local: exit [root @ ip-172-31-6-148 cdh-shell-bak] #

two。 Go to the Management-> Security interface of Cloudera Manager

3. Select "enable Kerberos" to enter the following interface

Ensure that all check items listed below have been completed

4. Click "continue" to configure relevant KDC information, including type, KDC server, KDC Realm, encryption type and update lifetime of the Service Principal (hdfs,yarn,hbase,hive, etc.) to be created.

5. Click "continue"

6. It is not recommended that Cloudera Manager manage krb5.conf. Click "continue".

7. Enter the Kerbers administrator account of Cloudera Manager, which must be the same as the account you created earlier, and click "continue"

8. Wait for Kerberos to be enabled, click "continue"

9. Click "continue"

10. Check to restart the cluster and click "continue"

11. Wait for the cluster to restart successfully, click "continue"

So far, Kerberos has been successfully enabled.

4.Kerberos usage

To run MapReduce tasks and operate Hive with fayson users, you need to create fayson users on all nodes in the cluster.

1. Create a principal for fayson using kadmin

[root@ip-172-31-6-148cdh-shell-bak] # kadmin.localAuthenticating as principal admin/admin@FAYSON.COM with password.kadmin.local: addprinc fayson@FAYSON.COMWARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policyEnter password for principal "fayson@FAYSON.COM": Re-enter password for principal "fayson@FAYSON.COM": Principal "fayson@FAYSON.COM" created.kadmin.local: exit [root @ ip-172-31-6-148cdh-shell-bak] #

two。 Log in to Kerberos using the fayson user

[root@ip-172-31-6-148cdh-shell-bak] # kinit faysonPassword for fayson@FAYSON.COM: [root@ip-172-31-6-148cdh-shell-bak] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: fayson@FAYSON.COMValid starting Expires Service principal09/05/17 17:19:08 09fayson@FAYSON.COMValid starting Expires Service principal09/05/17 * COM@FAYSON.COM renew until 09 + 12 cdh-shell-bak 17 17:19: 08 [root @ root-31-6-148root] #

3. Run a MapReduce job

[root@ip-172-31-6-148~] # hadoop jar / opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 10 1...Starting Job17/09/02 20:10:43 INFO mapreduce.Job: Running job: job_1504383005209_000117/09/02 20:10:56 INFO mapreduce.Job: Job job_1504383005209_0001 running in ubermode: false17/09/02 20:10:56 INFO mapreduce.Job: map0% reduce 0pea 09Unip 02 20: 11:09 INFO mapreduce.Job: map20% reduce 0/09/02 20:11:12 INFO mapreduce.Job: map40% reduce 0/09/02 20:11:13 INFO mapreduce.Job: map50% reduce 0/09/02 20:11:15 INFO mapreduce.Job: map60% reduce 0/09/02 20:11:16 INFO mapreduce.Job: map70% reduce 0/09/02 20:11:19 INFO mapreduce.Job: map80% reduce 0/09/02 20:11:21 INFO mapreduce. Job: map100% reduce 0/09/02 20:11:26 INFO mapreduce.Job: map100% reduce 100-09-02 20:11:26 INFO mapreduce.Job: Job job_1504383005209_0001 completedsuccessfully

4. Use beeline to connect to hive for testing

[root@ip-172-31-6-148cdh-shell-bak] # beeline Beeline version 1.1.0-cdh6.12.1 by Apache Hivebeeline >! connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148.fayson.com@FAYSON.COM...Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://localhost:10000/ > show tables ... INFO: OK+-+--+ | tab_name | +-+-+ | test_table | +-+-- + 1 row selected (0.194 seconds) 0: jdbc:hive2://localhost:10000/ > select * from test_table ... INFO: OK+--+ | test_table.s1 | test_table.s2 | +-- + | 4 | lisi | | 1 | test | | 2 | fayson | | 3 | zhangsan | +-- + 4 rows selected (0.144 seconds) 0: jdbc:hive2://localhost:10000/ > |

Run a Hive MapReduce job

0: jdbc:hive2://localhost:10000/ > select count (*) from test_table;...INFO: OK+-+--+ | _ c0 | +-+-+ | 4 | +-+-+ 1 row selected (35.779 seconds) 0: jdbc:hive2://localhost:10000/ >

5. common problem

1. Running a MapReduce job with Kerberos user identity reported an error

Main: run as user is faysonmain: requested yarn user is faysonRequested user fayson is not whitelisted and has id 501,whichis below the minimum allowed 1000Failing this attempt. Failing the application.17/09/02 20:05:04 INFO mapreduce.Job: Counters: 0Job Finished in 6.184 secondsjava.io.FileNotFoundException: File does not exist:hdfs://ip-172-31-6-148:8020/user/fayson/QuasiMonteCarlo_1504382696029_1308422444/out/reduce-out at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall (DistributedFileSystem.java:1266) at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall (DistributedFileSystem.java:1258) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve ( FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus (DistributedFileSystem.java:1258) at org.apache.hadoop.io.SequenceFile$Reader. (SequenceFile.java:1820) at org.apache.hadoop.io.SequenceFile$Reader. (SequenceFile.java:1844) at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi (QuasiMonteCarlo.java:314) at org.apache.hadoop.examples.QuasiMonteCarlo.run (QuasiMonteCarlo.java:354) at org.apache.hadoop.util.ToolRunner.run (ToolRunner.java:70) at org .apache.hadoop.examples.QuasiMonteCarlo.main (QuasiMonteCarlo.java:363) at sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:606) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke (ProgramDriver.java:71) at org.apache.hadoop.util.ProgramDriver.run (ProgramDriver.java:144) at org. Apache.hadoop.examples.ExampleDriver.main (ExampleDriver.java:74) at sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethod) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:606) atorg.apache.hadoop.util.RunJar.run (RunJar.java:221) at org.apache.hadoop.util.RunJar.main (RunJar.java:136)

The reason for the problem is that Yarn restricts the submission of jobs by users whose id is less than 10000

Solution: modify the min.user.id of Yarn to solve the problem

At this point, the study on "how to enable Kerberos in the CDH cluster" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.