Shulou(Shulou.com)06/03 Report--

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

1. Purpose of document writing

This document focuses on how to install configuration and use Sentry in an Kerberos-enabled CDH cluster.

Content Overview

1. How to install the Sentry service

How 2.Hive/Impala/Hue/HDFS Services integrate with Sentry

3.Sentry test

Test environment

1. The operating system is CentOS6.5

Version 5.11.1 for 2.CM and CDH

3. Adopt root user operation

Precondition

The 1.CDH cluster is running normally

two。 Kerberos is enabled in the cluster and is in normal use

2.Sentry installation

1. Create a sentry database in MySQL

Build a table sentence:

Create database sentry default character set utf8

CREATE USER 'sentry'@'%' IDENTIFIED BY' password'

GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%'

FLUSH PRIVILEGES

Command line operations:

[root@ip-172-31-6-148527-hive-HIVEMETASTORE] # mysql-uroot-pEnter password:. MySQL > create database sentry default character set utf8;Query OK, 1 row affected (0.00 sec) mysql > CREATE USER 'sentry'@'%' IDENTIFIED BY' password';Query OK, 0 rows affected (0.00 sec) mysql > GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%';Query OK, 0 rows affected (0.00 sec) mysql > FLUSH PRIVILEGES;Query OK, 0 rows affected (0.00 sec) mysql >

two。 Go to the Cloudera Manager console and click "add Services"

3. Go to the service add interface, select Sentry service, and click "continue"

4. Select the installation node for Sentry Server and Gateway, and click continue

5. Enter the database information of the Sentry service, click test, test pass, click "continue"

6. Wait for the service to be installed successfully, click "continue"

7. Click "finish", and the Sentry service installation is complete.

3.Sentry configuration

3.1Hive configuration

1. Configure Hive to use Sentry services

two。 Turn off the user emulation function of Hive

3.2Impala configuration

Configure Impala to use Sentry

3.3Hue configuration

Configure Hue to use Sentry

3.4HDFS configuration

Configure HDFS to enable synchronization of ACLs and Sentry permissions

After completing the above configuration, go back to the Cloudera Manager home page, deploy the client configuration and restart the related services.

4.Sentry test

4.1Creating hive superuser

Log in to Kerberos using the hive user as follows

[root@ip-172-31-6-148196-hive-HIVEMETASTORE] # kinit-kt hive.keytab hive/ip-172-31-6-148.fayson.com@FAYSON.COM [root@ip-172-31-6-148196-hive-HIVEMETASTORE] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: hive/ip-172-31-6-148.fayson.com@FAYSON.COMValid starting Expires Service principal09/07/17 02:26:04 09max 08 / 17 02:26:04 krbtgt/FAYSON.COM@FAYSON.COM renew until 09 196-hive-HIVEMETASTORE 12 impulse 17 02:26: 04 [root @ root-31-6-14 196-hive-HIVEMETASTORE] #

1. Connect to HiveServer2 using beeline

[root@ip-172-31-6-148196-hive-HIVEMETASTORE] # beeline Beeline version 1.1.0-cdh6.12.1 by Apache Hivebeeline >! connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148@FAYSON.COMscan complete in 3msConnecting to jdbc:hive2://localhost:10000/ Principal=hive/ip-172-31-6-148@FAYSON.COMConnected to: Apache Hive (version 1.1.0-cdh6.12.1) Driver: Hive JDBC (version 1.1.0-cdh6.12.1) Transaction isolation: TRANSACTION_REPEATABLE_READ0: jdbc:hive2://localhost:10000/ >

two。 Create an admin role

0: jdbc:hive2://localhost:10000/ > create role admin;...INFO: OKNo rows affected (0.37 seconds) 0: jdbc:hive2://localhost:10000/ >

3. Give administrator permissions to the admin role

0: jdbc:hive2://localhost:10000 > grant all on server server1 to role admin

...

INFO: OK

No rows affected (0.221 seconds)

0: jdbc:hive2://localhost:10000 >

4. Authorize the admin role to the hive user group

0: jdbc:hive2://localhost:10000 > grant role admin to group hive

...

INFO: OK

No rows affected (0.162 seconds)

0: jdbc:hive2://localhost:10000 >

The above creates an admin role:

Admin: with administrator privileges, you can read and write all databases and authorize the hive group (the corresponding operating system group)

4.2 create a test table

Log in to Kerberos using the hive user, log in to HiveServer2 through beeline, create the test table, and insert test data

0: jdbc:hive2://localhost:10000 > create tabletest (S1 string, S2 string) row format delimited fields terminated by','

...

INFO: OK

No rows affected (0.592 seconds)

0: jdbc:hive2://localhost:10000 > insert into test values ('axiomagem dongb'), (' 1hmpdl')

...

INFO: OK

No rows affected (20.123 seconds)

0: jdbc:hive2://localhost:10000 >

4.3 create a test role and authorize the role to the user group

Create two roles:

Read: only the default library test table can be read and authorized to the fayson user group

Write: only default library test table can be written and authorized to user_w user group

Note: fayson and user_w users must exist on all nodes in the cluster, the user default user group is the same as the user name, and the authorization is for the user group, not the user.

[root@ip-172-31-6-148cdh-shell-master] # id faysonuid=501 (fayson) gid=501 (fayson) groups=501 (fayson) [root@ip-172-31-6-148cdh-shell-master] # useradd user_ w [root @ ip-172-31-6-148cdh-shell-master] # id user_wuid=502 (user_w) gid=502 (user_w) groups=502 (user_w) [root@ip-172-31-6-148cdh-shell-master] #

1. Use the hive user to create read and write roles, and authorize the read role to have select permissions on the test table and the write role to have insert permissions on the test table

0: jdbc:hive2://localhost:10000 > create role read

...

INFO: OK

No rows affected (0.094 seconds)

0: jdbc:hive2://localhost:10000 > grant select on table test torole read

...

INFO: OK

No rows affected (0.1 seconds)

0: jdbc:hive2://localhost:10000 > create role write

...

INFO: OK

No rows affected (0.105 seconds)

0: jdbc:hive2://localhost:10000 > grant insert on table test to role write

...

INFO: OK

No rows affected (0.112 seconds)

0: jdbc:hive2://localhost:10000 >

two。 Authorize the read role to the fayson user group and the write role to the user_w user group

0: jdbc:hive2://localhost:10000 > grant role read to group fayson;...INFO: OKNo rows affected (0.187 seconds) 0: jdbc:hive2://localhost:10000 > grant role write to group user_w;...INFO: OKNo rows affected (0.101 seconds) 0: jdbc:hive2://localhost:10000 >

3. Create fayson and user_w users using kadmin

# kadmin.localAuthenticating as principal hive/admin@FAYSON.COM with password.kadmin.local: addprinc fayson@FAYSON.COMWARNING: no policy specified for fayson@FAYSON.COM; defaulting to no policyEnter password for principal "fayson@FAYSON.COM": Re-enter password for principal "fayson@FAYSON.COM": Principal "fayson@FAYSON.COM" created.kadmin.local: addprinc user_w@FAYSON.COMWARNING: no policy specified for user_w@FAYSON.COM Defaulting to no policyEnter password for principal "user_w@FAYSON.COM": Re-enter password for principal "user_w@FAYSON.COM": Principal "user_w@FAYSON.COM" created.kadmin.local:

4.4beeline verification

1. Log in to Kerberos using the fayson user

[root@ip-172-31-6-14814] # kinit faysonPassword for fayson@FAYSON.COM: [root@ip-172-31-6-14814] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: fayson@FAYSON.COMValid starting Expires Service principal09/07/17 02:48:35 09 krbtgt/FAYSON.COM@FAYSON.COM renew until krbtgt/FAYSON.COM@FAYSON.COM renew until 09 / 14 to 17 02:48: 35 [root @ ip-172-31-6-14814] #

Verify by connecting to HiveServer2 through beeline

[root@ip-172-31-6-148C] # beeline Beeline version 1.1.0-cdh6.12.1 by Apache Hivebeeline >! connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148.fayson.com@FAYSON.COM...0: jdbc:hive2://localhost:10000/ > show tables ... INFO: OK+-+--+ | tab_name | +-+-+ | test | +-+-- + 1 row selected (0.403 seconds) 0: jdbc:hive2://localhost:10000/ > select * from test INFO: OK+--+ | test.s1 | test.s2 | +-- + | a | b | 1 | 2 | 11122 | a | b | 1 | 2 | | 333 | 5555 | | eeee | dddd | +-+ 7 rows selected (0.282 seconds) 0: jdbc:hive2://localhost:10000/ > insert into test values ("2") "222") Error: Error while compiling statement: FAILED: SemanticException No valid privileges User fayson does not have privileges for QUERY The required privileges: Server=server1- > Db=default- > Table=test- > action=insert; (state=42000,code=40000) 0: jdbc:hive2://localhost:10000/ >

Perform MapReduce tasks for Hive

0: jdbc:hive2://localhost:10000/ > select count (*) from test;...INFO: OK+-+--+ | _ c0 | +-+-+ | 7 | +-+-- + 1 row selected (30.688 seconds) 0: jdbc:hive2://localhost:10000/ >

two。 Log in to Kerberos using the user_w user

[root@ip-172-31-6-14814] # kinit user_wPassword for user_w@FAYSON.COM: [root@ip-172-31-6-14814] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: user_w@FAYSON.COMValid starting Expires Service principal09/07/17 03:01:56 09 ip-172 krbtgt/FAYSON.COM@FAYSON.COM 17 03:01:56 Renew until 09 take 14 shock 17 03:01: 56 [root @ ip-172-31-6-14814] #

Log in to HiveServer2 authentication using beeline

[root@ip-172-31-6-148C] # beeline Beeline version 1.1.0-cdh6.12.1 by Apache Hivebeeline >! connect jdbc:hive2://localhost:10000/;principal=hive/ip-172-31-6-148.fayson.com@FAYSON.COM...0: jdbc:hive2://localhost:10000/ > show tables INFO: OK+-+--+ | tab_name | +-+-+ | test | +-+-- + 1 row selected (0.343 seconds) 0: jdbc:hive2://localhost:10000/ > select * from test;Error: Error while compiling statement: FAILED: SemanticException No valid privileges User user_w does not have privileges for QUERY The required privileges: Server=server1- > Db=default- > Table=test- > Column=s1- > action=select (state=42000,code=40000) 0: jdbc:hive2://localhost:10000/ > insert into test values ("2", "333");... INFO: OKNo rows affected (19.379 seconds) 0: jdbc:hive2://localhost:10000/ >

Verification summary:

The group to which the fayson user belongs has read permission for test table for fayson. Only select and count operations can be performed on test table, not insert operation.

The group to which the user_w user belongs has permission to write to the test table for user_w. Only insert operations can be performed on the test table, not select operations.

4.5HDFS verification

1. Log in to Kerberos using the fayson user and do the following

Use the HDFS command to cat, ls, put, and so on / user/hive/warehouse/test

[root@ip-172-31-6-148C] # hadoop fs-ls / user/hive/warehousels: Permission denied: user=fayson, access=READ_EXECUTE Inode= "/ user/hive/warehouse": hive:hive:drwxrwx-- x [root @ ip-172-31-6-148x] # hadoop fs-ls / user/hive/warehouse/testFound 5 items-rwxrwx--x+ 3 hive hive 8 2017-09-05 12:52 / user/hive/warehouse/test/000000_0-rwxrwx--x+ 3 hive hive 8 2017-09-05 13:44 / user/hive/warehouse/test/000000_0_copy_1-rwxrwx- -x + 3 hive hive 8 2017-09-07 02:36 / user/hive/warehouse/test/000000_0_copy_2-rwxrwx--x+ 3 hive hive 6 2017-09-07 03:04 / user/hive/warehouse/test/000000_0_copy_3-rwxrwx--x+ 3 hive hive 19 2017-09-05 13:01 / user/hive/warehouse/test/ test.txt [root @ ip-172-31-6-14814] # hadoop Fs-cat / user/hive/warehouse/test/test.txt333,5555eeee DDD [root @ ip-172-31-6-148] # hadoop fs-rm / user/hive/warehouse/test/test.txtrm: Failed to move to trash: hdfs://ip-172-31-6-148.fayson.com:8020/user/hive/warehouse/test/test.txt: Permission denied: user=fayson, access=WRITE Inode= "/ user/hive/warehouse/test": hive:hive:drwxrwx-- x [root @ ip-172-31-6-148x] # hadoop fs-put a.txt / user/hive/warehouse/test/put: Permission denied: user=fayson, access=WRITE, inode= "/ user/hive/warehouse/test": hive:hive:drwxrwx-- x [root @ ip-172-31-6-148x] #

two。 Log in to Kerberos using the user_w user and do the following

# kinit user_wPassword for user_w@FAYSON.COM: [root@ip-172-31-6-root@ip-172] # hadoop fs-ls / user/hive/warehousels: Permission denied: user=user_w, access=READ_EXECUTE Inode= "/ user/hive/warehouse": hive:hive:drwxrwx-- x [root @ ip-172-31-6-148x] # root-ls / user/hive/warehouse/testls: Permission denied: user=user_w, access=READ_EXECUTE, inode= "/ user/hive/warehouse/test": hive:hive:drwxrwx-- x [root @ ip-172-31-6-148x] # hadoop fs-cat / user/hive/warehouse/test/test.txtcat: Permission denied: user=user_w, access=READ Inode= "/ user/hive/warehouse/test/test.txt": hive:hive:-rwxrwx-- x [root @ ip-172-31-6-14848] # hadoop fs-rm / user/hive/warehouse/test/test.txt17/09/07 03:21:21 INFO fs.TrashPolicyDefault: Moved: 'hdfs://ip-172-31-6-148.fayson. Com 6-148.fayson.com:8020/user/user_w/.Trash/Current/user/hive/warehouse/test/ test.txt [root @ ip-172-31-6-148t] # hadoop fs-put a.txt / user/hive/warehouse/test/ [root @ ip-172-31-6-148t] #

The group to which the fayson user belongs is fayson, which has the read permission of the test table. You can browse and view the contents of the files in the data directory (/ user/hive/warehouse/test) of the test table. You cannot delete the files under the / test/ directory or put the files to the test directory.

The user_w user belongs to user_w, has the write permission of the test table, and can operate on the data directory of the test table, put files and delete data files, but cannot browse and view the contents of the files in the directory.

It shows that Sentry realizes the ACL synchronization of HDFS.

4.6Hue verification

1. Administrators using Hue, adding test users fayson and user_w for Hue

two。 Use the fayson user to log in to Hue and verify read permissions

You can view test table data

You can perform Count operations

Cannot Insert operation

File Browser browsing

Cannot browse parent directory / user/hive/warehouse

You can browse the data directory / user/hive/warehouse/test of the test table

You can view the contents of the files in the / user/hive/warehouse/test directory

Cannot modify data files in / user/hive/warehouse/test directory

3. Use the user_w user to log in to Hue and verify write permissions

Cannot view test table

Count operation is not allowed

You can insert data into a test table

FileBrowser operation

Cannot browse parent directory / user/hive/warehouse

Cannot browse the data directory / user/hive/warehouse/test of the test table

Both fayson and user_w users can see the test table through the hue interface, and the fayson user group with the read role can select and count the test, and browse and view the data directory / user/hive/warehouse/test of the test table through File Browser. User_w user groups with the write role can only insert the test table, but cannot browse and view the data directory / user/hive/warehouse/test of the test table through File Browser. Indicates that the operation and authorization of Sentry on the command line is still valid in Hue.

4.7Impala verification

1. Log in to Kerberos using the fayson user

[root@ip-172-31-6-14814] # kinit faysonPassword for fayson@FAYSON.COM: [root@ip-172-31-6-14814] # klistTicket cache: FILE:/tmp/krb5cc_0Default principal: fayson@FAYSON.COMValid starting Expires Service principal09/07/17 06:36:05 08 krbtgt/FAYSON.COM@FAYSON.COM renew until 17 06:36:05 krbtgt/FAYSON.COM@FAYSON.COM renew until 09 Chara 14 06:36: 05 [root @ ip-172-31 # impala-shell Starting Impala Shell without Kerberos authentication...Connected to ip-172-31-9-33.fayson.com:21000Server version: impalad version 2.9.0-cdh6.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9) [ip-172-31-9-33.fayson.com:21000] > show tables Query: show tables+-+ | name | +-+ | test | +-+ Fetched 1 row (s) in 0.02s [IP-172-31-9-33.fayson.com:21000] > select * from test . +-+-+ | S1 | S2 | +-+-+ | testaa | testbbb | | 11122 | | 2323 | | 2 | 333 | a | b | 1 | 2 | test | 2 | fayson | | 3 | zhangsan | | a | | | b | | 1 | 2 | +-+-+ Fetched 11 row (s) in 0.19s [IP-172-31-9-33.fayson.com:21000] > select count (*) from test | . +-+ | count (*) | +-+ | 11 | +-+ Fetched 1 row (s) in 0.14s [IP-172-31-9-33.fayson.com:21000] > insert into test values ('test44','test55') Query: insert into test values ('test44','test55') Query submitted at: 2017-09-07 06:37:00 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)ERROR: AuthorizationException: User' fayson@FAYSON.COM' does not have privileges to execute 'INSERT' on: default.test [IP-17231-9-33.fayson.com:21000] >

two。 Log in to Kerberos using the user_w user

Log in to Impala-shell to operate

[root@ip-172-31-6-148C] # impala-shell... * Welcome to the Impala shell. (Impala Shell v2.9.0-cdh6 .12.1 (5131a03) built on Thu Aug 24 09:27:32 PDT 2017) * * [Not connected] > connect ip-172-31-9-33.fayson.com:21000 Connected to ip-172-31-9-33.fayson.com:21000Server version: impalad version 2.9.0-cdh6.12.1 RELEASE (build 5131a031f4aa38c1e50c430373c55ca53e0517b9) [ip-172-31-9-33.fayson.com:21000] > show tables;Query: show tables+-+ | name | +-+ | test | +-+ Fetched 1 row (s) in 0.29s [IP-31-9-33.fayson.com:21000] > select * from test Query: select * from testQuery submitted at: 2017-09-07 06:31:23 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)ERROR: AuthorizationException: User 'user_w@FAYSON.COM' does not have privileges to execute' SELECT' on: default.test [ip-172-31-9-33.fayson.com:21000] > insert into test values Query: insert into test values Query submitted at: 2017-09-07 06:32:07 (Coordinator: http://ip-172-31-9-33.fayson.com:25000)Query progress can be monitored at: http://ip-172-31-9-33.fayson.com:25000/query_plan?query_id=ec406e621c7534c7:6bcbbd5300000000Modified 1 row (s) in 0.63s [IP-17231-9-33.fayson.com:21000] >

Verification summary:

After the integration of Impala and Sentry, Sentry can be used for rights management. Fayson user groups with the role of read can only perform select and count operations on the test table, not insert data, and user_w with the role of write

User groups can only insert data on the test table, not select and count operations. It shows that Sentry realizes the synchronization of Hive permissions and Impala.

Drunken whips are famous horses, and teenagers are so pompous! Lingnan Huan Xisha, under the vomiting liquor store! The best friend refuses to let go, the flower of data play!

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

It is recommended to follow Hadoop practice, the first time, share more Hadoop practical information, welcome to forward and share.

Original article, welcome to reprint, reprint please indicate: reproduced from the official account of Wechat Hadoop

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.