Shulou(Shulou.com)06/02 Report--

This article mainly explains the problem of "Hive failed to create UDF in SQL standard permission mode". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "Hive failed to create UDF in SQL standard permission mode".

Environment:

CDH 5.16

Hive 1.1.0

Kerberos enabled

Hive authorizes the use of SQL StandardsBased Authorization mode (hereinafter referred to as SSBA mode)

Symptoms:

After compiling the jar package for UDF, upload it to the HDFS directory.

Hdfs dfs-mkdir / udfhdfs dfs-put-f my_udf.jar / udf

Enter the beeline client as administrator user hive.

Switch to the administrator role and execute:

Set role admin

When prompted to successfully switch roles, execute the create custom function statement:

Create function default.ch_cnv as' com.my.hive.udf.ChsUDF' using jar' hdfs:///udf/my_udf.jar'

Create a custom function to report an error, indicating that there is no "administrative permission" for the DFS_URI object applied to the CREATE FUNCTION:

Error: Error while compiling statement:FAILED: HiveAccessControlException Permission denied: Principal [name=hive,type=USER] does not have following privileges for operation CREATEFUNCTION [[ADMINPRIVILEGE] on Object [type=DFS_URI, name=hdfs://nameservice1/udf/my_udf.jar]] (state=42000,code=40000)

When you try to create a temporary function with temporary, the same error is reported:

Create temporary function default.ch_cnv as'com.my.hive.udf.ChsUDF' using jar' hdfs:///udf/my_udf.jar'

Analysis ideas:

In SSBA mode, executing some statements requires that the corresponding HDFS URI must be the permission of the rwx+ owner, which is very tedious (so starting from Hive 3.0.0, when authorization is based on SSBA, the URI permission requirements have been greatly modified. For example, when creating an appearance, the user is no longer required to be the owner of the HDFS directory, only to have rw permissions). Could there be a problem here?

Check the official documents first:

Https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization

The official document clearly shows that the CREATE FUNCTION operation has nothing to do with HDFS URI permissions.

Is the official document written wrong?

Try to change the jar permission to 777 and change the user and group to hive.

Retry the create function statement with no effect and still report the same error.

If this road is impassable, let's see if we can start with other information.

Re-enter beeline as hive, do not execute set role admin, execute create function statement as ordinary public role, and prompt:

Error: Error while compiling statement:FAILED: HiveAccessControlException Permission denied: Principal [name=hive,type=USER] does not have following privileges for operation CREATEFUNCTION [[ADMINPRIVILEGE] onObject [type=DATABASE, name=default], [ADMINPRIVILEGE] on object [type = DFS_URI, name=hdfs://nameservice1/udf/my_udf.jar], [ADMINPRIVILEGE] onObject [type=FUNCTION, name=default.ch_cnv]] (state=42000, code=40000)

At this point, you can see that three permissions are required to execute create function:

1. Admin permissions of the database

2. Admin permission of HDFS URI

3. Admin permission of function object

Among them, 1 and 3 are exactly the same as the description of the official document. After we set role admin, both 1 and 3 are satisfied. Only 2 is still not satisfied, which means that there is not a problem with our administrator role configuration or administrator role switching.

There is only one possibility left. This function of Hive has BUG.....

During the search, I found that a netizen encountered the same problem:

Http://lxw1234.com/archives/2016/01/600.htm

Although no one has answered this question, it at least shows that this is not a strange case.

Go to the official JIRA of Apache and search. Sure enough, there are relevant BUG reports:

Https://issues.apache.org/jira/browse/HIVE-11175

Judging from the description and the patch submitted, it is Hive who mistakenly treats an input resource (UDF's JAR package) as an output resource when performing semantic analysis of create function in SSBA mode. Administrator permission to write to the path is required (it should actually be read permission), and an existing input resource is read-only, resulting in permanently stuck on this validation.

Solution:

Since the SSBA pattern has been implemented for the Hive cluster, it is certainly impossible for us to temporarily turn off the validation of the cluster in order to load a UDF function. A complete shutdown will be exposed to security risks for a period of time, and the service needs to be restarted. Temporary shutdown requires a restart, and startup requires a restart, which will be very troublesome.

Another way is to apply the patches submitted in the previous community to the source code to recompile, and then replace the problematic JAR package. This patch has not yet been integrated into the main line of Hive, so there is some risk in doing so. And at present, this cluster is a commercial version of CDH purchased by customers, and it is not recommended to do it yourself.

What shall I do? It's time to invite the oldest Hive CLI to play!

It is precisely because Hive CLI is not compatible with the complete authorization mode (Sentry/Ranger/SSBA) that it can bypass these security restrictions, so it is officially replaced by beeline, only in versions prior to Hive3, as a compatibility feature.

So once Hive authorization is enabled, it is recommended to disable Hive CLI for ordinary users (unless you have the need for Spark multi-tenancy, the following HMS proxy will be disabled). The most common prohibition method is to configure Hive's hadoop.proxyuser.hive.groups (Hive Metastore access control and proxy user group override) to change the specified value from * to a specific administrator group, so that ordinary users cannot operate even if they enter the Hive CLI, but the administrator group users can still enter the Hive CLI operation.

So we can use the hive administrator user to go directly to Hive CLI and bypass the verification process of SSBA to create the UDF function:

Kinit-k-t hive.keytab hive@XXX.COMhive

After entering the Hive CLI, execute:

Create function default.ch_cnv as' com.my.hive.udf.ChsUDF' using jar' hdfs:///udf/my_udf.jar'

Prompt to create successfully!

Switch to beeline and try the HQL statement with the UDF function, no problem. Got it!

Thank you for your reading, the above is the content of "Hive failed to create UDF in SQL standard permission mode". After the study of this article, I believe you have a deeper understanding of the problem of Hive failed to create UDF in SQL standard permission mode, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.