Shulou(Shulou.com)06/03 Report--

The cdh version used by the company is 5.14. The steps for integrating Sentry are simple and can be completed by referring to the official documentation:

Https://www.cloudera.com/documentation/enterprise/5-14-x/topics/sentry.html

This article is mainly about the construction of openldap and the use of phpldap, as well as the integration of ldap and (hive,impala,hue).

The specific concept of ldap can refer to this article.

Introduction to the concept and principle of LDAP

one。 After installing OpenLDAP yum install openldap openldap-clients openldap-servers, directly launch ```service slapd start ```to set the administrator password ```slappasswd-h {SSHA}``` and then ask you to enter a plaintext password, return you an encrypted password, and remember the returned password.

Use the ldapsearch command to query the administrator's dn:

Ldapsearch-LLL-Y EXTERNAL-H ldapi:///-b cn=config "olcRootDN=cn=Manager,dc=my-domain,dc=com" dn olcRootDN olcRootPW

The dn returned here is olcDatabase= {2} hdb,cn=config, and the password and other information is also returned. What is useful here is the encryption method of dn and password, for example, here SSHA,dn is the Entry DN modified later, and if you know the encryption method of password, you can use this encryption method to generate a new password.

Use ldapmodify to modify entries

Vim chrootpw.ldif # this is the first step to get the administrator dndn: olcDatabase= {2} hdb,cn=configchangetype: modifyreplace: olcSuffix# the suffix olcSuffix: dc=xinniu,dc=com#olcSuffix of the domain name you want to change must be added, otherwise an error will occur after modification Later, restart openldap will fail dn: olcDatabase= {2} hdb,cn=configchangetype: modifyreplace: olcRootDN# the domain name you want to change to olcRootDN: cn=admin,dc=xinniu,dc=comdn: olcDatabase= {2} hdb,cn=configchangetype: modifyreplace: olcRootPW# here is the encrypted password olcRootPW: {SSHA} uYnICxla0NrUC5b/ha4i1JeOTCoUchV+ saved just now

As shown in the figure:

Use the following command to modify the administrator entry:

Ldapmodify-Y EXTERNAL-H ldapi:///-f chrootpw.ldif

If it appears:

Modifying entry "olcDatabase= {2} bdb,cn=config"

Restart the service for the changes to take effect

/ etc/init.d/slapd restart

Import basic data structures

We need to import some basic Schema into LDAP. These Schema files are located in the / etc/openldap/schema/ directory, and schema controls which object classes and attributes the item has.

Ldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/cosine.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/nis.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/collective.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/corba.ldifldapadd-Y EXTERNAL -H ldapi:///-D "cn=config"-f / etc/openldap/schema/core.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/duaconf.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/dyngroup.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/inetorgperson.ldifldapadd-Y EXTERNAL-H ldapi: / /-D "cn=config"-f / etc/openldap/schema/java.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/misc.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/openldap.ldifldapadd-Y EXTERNAL-H ldapi:///-D "cn=config"-f / etc/openldap/schema/pmi.ldifldapadd-Y EXTERNAL-H ldapi:/// -D "cn=config"-f / etc/openldap/schema/ppolicy.ldif

At this point, the administrator's password is set. You can use the client to manage openldap.

Plan to use phpldapadmin to manage openldap, this tool uses more people, online documents, problems are easy to solve.

two。 Install phpldapadmin (ldap management tools)

First install Apache and PHP:

Yum-y install httpd php php-ldap php-gd php-mbstring php-pear php-bcmath php-xml

Then install phpldapadmin:

Yum-y install phpldapadmin

Httpd integrates with phpldapadmin:

Modify the configuration file:

[root@localhost ~] # vim / etc/phpldapadmin/config.php#397 line uncomment, line 398 add comment $servers- > setValue ('login','attr','dn'); / / $servers- > setValue (' login','attr','uid')

It needs to be modified here. If it is not modified, phpldapadmin will use uid to search for entries, but will not use dn to search, resulting in a password error.

The ip filtering rule of [root@localhost ~] # phpldapadmin denies all ip access by default. Modify the rule before we can access it.

Vim / etc/httpd/conf.d/phpldapadmin.conf## this is the Apache2.2 and previous modification method / / modify the configuration to change the following Deny from all to Allow from all## this is the apache2.4 and later modification method # allow 172.16.10.16 access to Require ip 172.16.10.allow the network segment 172.16 to access Require ip 172.16.0.0max allow all ip to access Require all We can modify granted according to our needs and launch it after modification: service httpd restart interface:

Landing

User name, default is cn=Manager,dc=my-domain,dc=com (this can also be changed, follow the steps to change the password)

Enter the password you just set.

After logging in, my management interface reports this information:

This base cannot be created with PLA

The display results are as follows:

After querying, the reason is that the root node needs to be initialized before it can be used. The final processing is as follows:

1. Create an initroot.ldif file to prepare for the creation and initialization of the root node, as shown below:

Dn: dc=xinniu,dc=com

# suffix of domain name

O: ldap

Objectclass: dcObject

Objectclass: organization

2. Execute ldapadd-f initroot.ldif-x-D cn=admin,dc=xinniu,dc=com-W after entering the password of the previously configured OpenLDAP, you can complete the creation of the LDAP root node, as shown in the following figure:

After completing the above operations, go back to the phpldapadmin page, and you can see that the root node has appeared normally, and you can create related OU, Group, Account and other objects, as shown in the following figure:

three。 Integrate ldap and hive,impala,hue

First use phpldapadmin to create a user in ldap to test:

First create an ou called users

Then create a user under users: jiandan

When creating a new entry, select Generic User Account, and the entry cn=jiandan will appear after it is created.

Then click the entry cn=jiandan, click rename to change cn=jiandan to uid=jiandan, because hive will search for uid by default in ldap. If you use cn, you also need to configure a custom attribute in the advanced configuration.

1. First, integrate hive:

Modify the configuration in cdh:

This picture belongs to someone else. If you follow my configuration, BaseDN will write:

Ou=users,dc=my-domain,dc=com

Then test it with beeline

This is verified by ldap. If the password is incorrect, it will:

Later, there is a problem when hive integrates ldap in production. Neither hue nor impala will appear, which may occur when using LDAP from other departments instead of self-built LDAP.

That is the hive.server2.authentication.ldap.baseDN configuration item. When hive authenticates a user in ldap, it matches the user through uid, such as uid=xiaohuang,ou=users,dc=my-domain,dc=com. The problem is that sometimes ldap is not necessarily built by our department itself, but built by the operation and maintenance side of the technology center. When they create a user entry in LDAP, they do not give the entry a uid attribute, but build a CN attribute, such as: cn=xiaohuang,ou=users,dc=my-domain,dc=com, we use the configuration of the above document, it will not be found, resulting in verification failure.

What should I do at this time? Ask them to help us rebuild the item we want? They definitely don't have birds.

But did I just give up? It's impossible. Find this article by going to the official document.

Https://cwiki.apache.org/confluence/display/Hive/User+and+Group+Filter+Support+with+LDAP+Atn+Provider+in+HiveServer2#UserandGroupFilterSupportwithLDAPAtnProviderinHiveServer2-hive.server2.authentication.ldap.userDNPattern

See that there is an attribute, it can also make hive and imapla,hue, through wildcards to set, let hive go to LDAP to find the specified entry.

This property is to be specified in the

Hive-site.xml Hive service advanced configuration code snippet (safety valve)

Configuration in

Hive.server2.authentication.ldap.userDNPatterncn=%s,ou=users,dc=my-domain,dc=comtrue

This allows hive to search for entries at the beginning of CN instead of UID when doing account verification.

. The problem was solved cleverly.

two。 Then integrate impala:

Modify the configuration, similar to hive:

However, we need to add a parameter to the high-level code snippet, because our password is plaintext transmission, otherwise imapala will not start.

After restarting impala, let's test

I found that I had already ok.

3. Finally, let's integrate hue.

First, you need to configure these

After this configuration, we can log in to hue, and we can see that there is a ldap in the login interface.

Login is ready, but some errors will be reported after logging in:

[file]

We need to configure some additional parameters so that hue users can represent other users

1. Create a hue user in ldap

These are included in 2.hue_safety_valve.ini 's Hue service advanced configuration code snippet (safety valve)

[beeswax] close_queries=Trueuse_sasl=Falseauth_username=hueauth_password= xxxxxxx[impala] server_host=datanode1server_interface=hiveserver2server_port=21050query_timeout_s=100impersonation_enabled=Trueauth_username=hueauth_password=xxxxxxx

Add the following configuration to the Hive service advanced configuration code snippet (relief valve) of core-site.xml for hive and I mpala:

Impala's Impala command line parameter advanced configuration code snippet (relief valve) plus

-- authorized_proxy_user_config=hue=*

And then you won't report this mistake.

The above basically completed the integration of ldap and hive impala hue.

4. Hadoop user-> group mapping relationship integration ldap1. Principle

In the use of sentry, we find that the authorization of sentry is based on roles, and roles are assigned through groups. In the actual use of hive/impala, we use a user, how does hive know what permissions it has? In fact, each permission operation is completed through a mapping relationship such as user-> group-> role. The mapping relationship of group-> role is managed through the sentry database:

Sentry manages the mapping relationship between group and role through the three libraries sentry_role,sentry_group,sentry_role_group_map.

And how is the mapping relationship of user-group realized?

There is a related option in hive. The user group information of hadoop is used by default, that is, the mapping of user- > group in hive is directly used by hdfs, and where does the mapping information of user- > group of hdfs come from?

When we open the configuration of hdfs, we find that:

With such a configuration item, the group information of hdfs is actually the group information of the local shell that is called by default!

two。 Problems and causes

After the details are clear, let's talk about the actual problem. If we use the default configuration, when we use hive and impala, we connect to hiveserver or impalad. If there is no corresponding user locally, then sentry cannot get the information of user- > group, so the whole link of permission authentication cannot be completed, and an error will be reported.

So every time we have a new colleague to join us, we need to create a corresponding user on each relevant node, which is actually not only troublesome, but also low (which is most important), even if you write batch scripts.

3. Solution then we can store the mapping of user- > group through ldap, so that hdfs does not have to look for this mapping locally. We first import the users and groups on the current system into ldap, we need to use a tool: migrationtools3.1 install migrationtoolsyum install-y migrationtools

. Modify migrate_common.ph

Vim / usr/share/migrationtools/migrate_common.phDefault DNS domainDEFAULT_MAIL_DOMAIN = "zpbigdata.com"; Default base DEFAULT_BASE = "dc=zpbigdata,dc=com"

Use pl script to generate / etc/passwd and / etc/group into a file format that LDAP can read

Import an etl user here to test

Cat / etc/passwd | grep etl > / tmp/passwd cat / etc/group | grep etl > / tmp/group/usr/share/migrationtools/migrate_base.pl > / tmp/base.ldif / usr/share/migrationtools/migrate_passwd.pl / tmp/passwd > / tmp/passwd.ldif / usr/share/migrationtools/migrate_group.pl / tmp/group > / tmp/group.ldif

Import files into LDAP

Ldapadd-x-D "cn=Manager,dc=zpbigdata,dc=com"-W-f / tmp/base.ldif ldapadd-x-D "cn=Manager,dc=zpbigdata,dc=com"-W-f / tmp/passwd.ldif ldapadd-x-D "cn=Manager,dc=zpbigdata,dc=com"-W-f / tmp/group.ldif

This imports the people and group information

3.2 modify the configuration of hdfs

Change the value of hadoop.security.group.mapping to org.apache.hadoop.security.LdapGroupsMapping

Then enter the relevant parameters of Ldap:

Roughly, there are so many configurations to fill in, for reference only. For specific values, please refer to the specific attributes of objects in group and people established in your ldap.

4. Verification

After the configuration is complete, use the following command to test:

Sudo-u hdfs hdfs groups xxx (username)

If the printed group value corresponds to what you configured in ldap, then it is done.

To add new group and user later, you just need to add them in ldap.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.