In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-16 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >
Share
Shulou(Shulou.com)05/31 Report--
This article is about how HDFS implements rights management. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.
Introduction to privilege management of 1 HDFS
The rights management of HDFS is divided into two parts:
Basic rights management similar to linux (coarse-grained)
There are three kinds of rights management methods for management objects: user, group and other.
User: the owner of a directory or file
Group: the group in which the above owner belongs
Other: a general term for other users
Privilege management in ACL mode (fine-grained)
Can be precisely controlled to a certain user, a certain group has the corresponding permissions
For the two ways, please see the figure below.
The basic permissions are like the rights management of directories or files in linux, which is coarse-grained and cannot be controlled to a certain user, such as T1. The permission management method of ACL is fine, as follows
User:t1:r-x
Indicates that user T1 has read and executable permissions (there is no concept of executable in HDFS, let's just call it this, the x permission here is the right to enter the directory and list the files or contents under the directory)
2 basic rights management of HDFS
What are the initial basic permissions to create a file or directory? Go through the following 2 processes:
2.1 default file or directory permissions
You can specify permissions when you create a file or directory. If not, the default permissions are used, as follows
Catalogue: 777
Document: 666
2.2 uMask configured by the application client
Note: this configuration can be configured by the client, that is, the client controls the permission to create a file or directory.
The role of umask:
It is the restriction on the basic permission of HDFS. For example, 022 indicates that there is no restriction on owner, 2 means that write permission is not allowed for group, and 2 means that write permission is not allowed for other.
The umask configured by the application client actually takes the permissions set by the user minus the above umask permissions to get the final permissions of the file or directory. If you create a directory, the default permission is 777, and then minus the umask permission 022 equals 755, that is, by default, owner has read-write and executable permissions, the group where owner resides has read and executable permissions, and other has read and executable permissions
The following focuses on the process of obtaining umask:
First try to get the fs.permissions.umask-mode attribute in the configuration file, such as 022, and then build the umask permission object
If the above attribute is not configured, get the dfs.umask attribute (outdated, it is recommended to use the first way). The umask in this way is in decimal form, such as the above 022=2x8+2=18, that is, the umask here should be configured to 18.
If the above attributes are not configured, 18 or 022 is used by default to build the umask object.
At this point, the above process is the permission to create a file or directory without ACL settings.
3 HDFS ACL privilege management 3.1 ACL privilege management model
AclEntry objects, such as
User:t1:r-x 、 group:g1:rwx 、 other::--x 、 mask::r-x
Default:user:t1:r-x 、 default:group:g1:r-x 、 default:other::r-x 、 default:mask::r-x
It contains four parts:
Public class AclEntry {private final AclEntryType type; private final String name; private final FsAction permission; private final AclEntryScope scope;}
Name: that is, the name, such as T1 and G1, can also be null, such as the mask::r-x above.
FsAction: permission information, with a total of the following permissions
NONE ("-"), EXECUTE ("--x"), WRITE ("- w -"), WRITE_EXECUTE ("- wx"), READ ("rmuri -"), READ_EXECUTE ("rmurx"), READ_WRITE ("rw-"), ALL ("rwx")
AclEntryScope: enumerated values, such as ACCESS and DEFAULT
ACCESS: indicates that the ACL is for this directory or file, and the prefix is omitted.
DEFAULT: exists only in the directory, prefixed with default:,. Once a subdirectory or file is created in that directory, the ACL of the subdirectory or file inherits that ACL permission. That is, if the parent directory contains a default:user:t1:r-x record, a user:t1:r-x record will be automatically added when creating a subdirectory or file.
AclEntryType: enumerated values, such as USER, GROUP, OTHER, MASK
USER: the user:t1:r-x above indicates that T1 is a user
GROUP: the group:g1:rwx above indicates that G1 is a group
OTHER: a general term for all remaining users as shown in the other::--x above
MASK: the above mask::r-x is mainly used for filtering, such as the group:g1:rwx above. Although this is the rwx permission, when determining the permission, you still have to and with mask::r-x, that is, the w permission is filtered, so the actual permission of group:g1:rwx is group:g1:r-x, as shown in the following figure.
In fact, mask only works on user and all group whose name value is not null. This will be explained in more detail later.
Basic permissions can also be converted into ACL permissions, such as basic permissions such as drwxr-xr-x, and corresponding ACL permissions such as
User::rwx corresponds to user permission
(group::r-x) corresponding group permissions
Mask::r-x corresponds to group permission
Other::r-x corresponds to other permission
The above modification of basic permissions or the corresponding ACL permissions will affect each other. There is one exception: the above group::r-x is created based on the group permission in the basic permission, but directly modifying the group::r-x permission in the ACL does not directly affect the group permission in the basic permission. This is also a pitfall, that is, the group::r-x permission is abandoned after it is created.
Active modification of mask3.2.1 mask values in ACCESS scope
The corresponding permission of mask in ACCESS scope is the group permission in the basic permissions of HDFS. As follows:
Therefore, you can change the group permission in the basic permissions in either of the following two ways:
Hdfs dfs-chmod 751
If the basic permission is changed to 751, then mask takes the group permission as 5, that is, the rmerx permission, as shown in the following figure
Directly modify the permission value in mask acl
For example, hdfs dfs-setfacl-m mask::--x / user/lg/acl, modified the mask value, and also modified the group permission in the basic permissions.
3.2.2 passive recalculation of mask values
When acl is added or deleted, the mask will be recalculated as follows:
All name are not the permissions of acl of null's user type and all acl of group type.
Examples are as follows:
At this point, the group permissions in the basic permissions will also change with the change of mask.
So far, let's sum up mask:
It is mainly designed to filter the permissions of other user and group types. If you take the initiative to set mask, it will play a certain filtering role.
But once you add or delete acl, the value of mask will be recalculated, that is to say, what you set before is useless, which means that mask is useless. When you inadvertently execute chmod modification permissions, it may cause other people, such as T1 users, to suddenly lose some permissions. Once a T2 user's acl permission is added, T1 user's permission will suddenly come back. This can be regarded as a pit. I don't know why it is designed like this. Welcome to discuss it.
3.3mask in DEFAULT scope
The mask in DEFAULT scope is basically similar to the mask in ACCESS scope and is used to filter permissions.
3.3.1 active modification of mask value
There are two ways to value mask in the above ACCESS scope, while there is only one way in DEFAULT scope, that is
Directly modify the permission value in mask acl
Such as hdfs dfs-setfacl-m default:mask::--x / user/lg/acl
After modification, the actual permission of the above default:user:t1:r-x is only-- x, that is, the result of performing an and operation with default mask.
3.3.2 mask value is recalculated
When acl is added or deleted, the mask will be recalculated as follows:
All name does not merge the permissions of acl of type user of null and all acl of type group. Not explained here, see mask recalculation in ACCESS scope.
3.4 process analysis
The previous introduction is some basic concepts and theories. Now let's take a look at what happens in the following processes.
3.4.1 permission verification process
Example: verify that user U1 has read access to the path / user/lg/acl
1 check whether U1 is the owner of the path
If it is owner, use the user permission in the basic permission to determine the decision directly. The code is as follows
2 iterate through the ACL permissions of all ACCESS scope in this path
If the current ACL type is user and the user name matches, such as user:u1:r-x, you also need to perform the and operation on the Rmerx permission and the group permission (that is, the mask permission) in the basic permission as the actual permission. If mask::--x, the actual permission changes to-x permission after the and operation, that is, there is still no read permission.
If the current ACL type is group and the groups to which the U1 user belongs contains the group, as above, you still need to and the ACL right and the group right (that is, the mask right) in the basic permission to determine the final permission.
The code is as follows:
3 once the user does not match in the above permissions, the other permission in the basic permissions is used to determine
Note that there is no matching meaning here: if there is a user:u1:--x permission, it matches to the user, but the ACL does not have read permission, and this part of the other permission will not be executed at this time.
4 if the relevant permissions have not been found, the user is determined to have no permissions.
3.4.2 create a file or directory
1 if no basic permissions are specified when creating a file or directory, the default basic permissions are used
Catalogue: 777
Document: 666
2 apply the umask permissions configured by the client on the basis of the above basic permissions. If the default is 022, the directory and file will become
Catalogue: 755
Document: 644
3 if the parent directory contains the ACL permission information of DEFAULT scope
Default:user::rwx and the user permissions in the basic permissions of the above file or directory perform and operations as user permissions in the final basic permissions default:mask::r-x and group permissions in the basic permissions above to perform and operations as group permissions in the final basic permissions default:other::--x and other permissions in the basic permissions of the file or directory to perform and operations as other permissions in the final basic permissions
Examples are as follows:
Other permissions in default are directly used as ACCESS permissions for subdirectories or files
The code is as follows:
4 if you are creating a subdirectory, copy all default permissions to the default permissions of the subdirectory as a subdirectory
5 then recalculate the basic permissions of the subfile or directory (because the above process may modify the basic permissions)
For code, see
3.4.3 modify ACL
The value of mask is recalculated when you add, delete or modify an ACL operation
ACL changes in ACCESS scope recalculate mask in ACCESS scope
ACL changes in DEFAULT scope recalculate mask in DEFAULT scope
The way mask is calculated is explained above.
Thank you for reading! This is the end of this article on "how to achieve rights management in HDFS". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.