Shulou(Shulou.com)05/31 Report--

This article mainly explains "how to achieve Hadoop security". The content in the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to achieve Hadoop security.

Hadoop security

HBase runs on Hadoop 0.20.x, and you can use the security features-- as long as you use these two versions, 0.20s and CDH3B3, and replace hadoop.jar.

1.3.1.3. Ssh

Ssh must be installed and sshd must be run so that Hadoop scripts can remotely control other Hadoop and Hbase processes. Ssh must be connected, no password can be logged in, the detailed method can be Google ("ssh passwordless login").

1.3.1.4. DNS

HBase uses the local hostname to get the IP address. Both forward and reverse DNS are possible.

If your machine has more than one interface, Hbase will use the primary interface that hostname points to.

If that's not enough, you can set up hbase.regionserver.dns.interface to specify the main interface. Of course, the configuration files for your entire cluster must be consistent, and each host uses the same network interface.

Another way is to set hbase.regionserver.dns.nameserver to specify nameserver, not with the system.

1.3.1.5. NTP

The clock of the cluster should be basically consistent. A little inconsistency can be tolerated, but large inconsistencies can lead to strange behavior. Run NTP or something to synchronize your time.

If you query or encounter a strange fault, you can check whether the system time is correct!

1.3.1.6. Ulimit and nproc

HBase is a database that uses many file handles at the same time. The default value of 1024 used by most linux systems is not satisfied, resulting in FAQ: Why do I see "java.io.IOException... (Too many open files)" in my logs? Abnormal. Such an exception may also occur.

2010-04-06 03 INFO org.apache.hadoop.hdfs.DFSClient 04 INFO org.apache.hadoop.hdfs.DFSClient 37542 INFO org.apache.hadoop.hdfs.DFSClient: Exception increateBlockOutputStream java.io.EOFException 2010-04-06 03 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-6935524980745310745_1391901

So you need to modify your maximum file handle limit. Can be set to 10k. You also need to modify the nproc of hbase users. If it is too low, it will cause OutOfMemoryError exception. To be clear, these two settings are for the operating system, not Hbase itself. A common mistake is that the user Hbase is running, and the user who sets the maximum value is not a user. When Hbase starts, the first log will now have ulimit information, so you'd better check it.

1.3.1.6.1. Set up ulimit on Ubuntu

If you are using Ubuntu, you can set it like this:

Add a line to the file / etc/security/limits.conf, such as:

Hadoop-nofile 32768

You can replace hadoop with your user running Hbase and Hadoop. If you use two users, you need two. And with nproc hard and soft limits. Such as:

Hadoop soft/hard nproc 32000

.

Add this line to / etc/pam.d/common-session:

Session required pam_limits.so

Otherwise, the configuration on / etc/security/limits.conf will not take effect.

Log out and log in again for these configurations to take effect!

1.3.1.7. Dfs.datanode.max.xcievers

A Hadoop HDFS Datanode has an upper limit for simultaneous processing of files. This parameter is called xcievers (the author of Hadoop misspelled the word). Before you load it, make sure that you have configured at least 4096 of the xceivers parameter in this file conf/hdfs-site.xml:

Dfs.datanode.max.xcievers 4096

For HDFS configuration changes, remember to restart.

Without this configuration, you may encounter strange failures. You will see xcievers exceeded in Datanode's log, but running it will report a missing blocks error. For example, 20:10:31 on 10-12-08 INFO hdfs.DFSClient: Could not obtain block blk_XXXXXXXXXXXXXXXXXXXXXX_YYYYYYYY from any node: java.io.IOException: No live nodes contain current block. Will get new block locations from namenode and retry... [5]

Thank you for reading, the above is the content of "how to achieve Hadoop security". After the study of this article, I believe you have a deeper understanding of how to achieve Hadoop security, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.