Shulou(Shulou.com)05/31 Report--

Editor to share with you what are the common parameters of HBase, I believe most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

1. Generic and master configuration

Hbase.rootdir

Default: file:///tmp/hbase-${user.name}/hbase

The data root directory of region server, which is used to persist HBase. For example, to represent the'/ hbase' directory in hdfs and namenode runs on port 8020 of debugo01, you need to set it to hdfs:// debugo01:8020/hbase. This is a project that must be set, and the default local file system / tmp can only be used in stand-alone mode.

Hbase.master.port

Default: 60000

The Master service port of HBase.

Hbase.cluster.distributed

Default: false

The operation mode of HBase. False is in stand-alone mode and true is in distributed mode. If false,HBase and Zookeeper run in the same JVM.

Hbase.tmp.dir

A local temporary directory. It is important to note that the default / tmp is emptied on restart.

Default: / tmp/hbase-$ {user.name}

Hbase.master.info.port

Default: 60010

The port of HBase Master web UI. Set to-1 to not run web UI.

Hbase.master.info.bindAddress

Default: 0.0.0.0

IP bound by HBase Master web UI

Hbase.master.dns.interface

When using DNS, the network interface name of the IP address that Master uses to report.

Default: default

Hbase.master.dns.nameserver

When using DNS, the DNS domain name or IP address of RegionServer, which Master uses to determine the domain name used for communication.

Default: default

Hbase.snapshot.enabled

Whether to enable the snapshot feature.

Default value: true

Hbase.balancer.period

The interval at which the Master executes the region balancer.

Default: 300000

Hbase.master.logcleaner.ttl

The maximum time that Hlog exists in the. oldlogdir folder will be cleaned up by the thread of Master.

Default: 600000

Hbase.master.logcleaner.plugins

A set of LogCleanerDelegat that the LogsCleaner service executes. Values are represented by text spaced by commas. These WAL/HLog cleaners are called sequentially. You can put the ones that are called first first. You can implement your own LogCleanerDelegat, add it to Classpath, and write down the full name of the class here. It is usually added before the default value.

Default: org.apache.hadoop.hbase.master.TimeToLiveLogCleaner

Hbase.rest.port

Port of HBase REST server

Default: 8080

Hbase.rest.readonly

Define the operation mode of REST server. Can be set to the following value: false: all HTTP requests are allowed-GET/PUT/POST/DELETE. True: only GET requests are allowed

Default: false

Hbase.coprocessor.master.classes

The name of the coprocessor class used by Master. Use comma separation in the case of multiple classes. For example, the use of org.apache.hadoop.hbase.security.access.AccessController provides security control capabilities.

Default: null.

Hbase.rpc.engine

The rpc engine class used by HBase. Org.apache.hadoop.hbase.ipc with security verification is generally used. SecureRpcEngine class

Default: null.

2. RegionServer correlation

Hbase.regionserver.port

Default: 60020

Port bound by HBase RegionServer

Hbase.regionserver.info.port

Default: 60030

The port bound by the HBase RegionServer web interface. Set to-1 to not run web UI

Hbase.regionserver.info.port.auto

Default: false

Whether RegionServer can search for a free port binding when hbase.regionsever.info.port is already occupied. Off by default.

Hbase.regionserver.info.bindAddress

Default: 0.0.0.0

The IP address of the HBase RegionServer web interface

Hbase.regionserver.class

Default: org.apache.hadoop.hbase.ipc.HRegionInterface

RegionServer interface class. The client will use it when connecting to the region server.

Hbase.regionserver.lease.period

Default: 60000

The client lease HRegion server period, that is, the timeout threshold. The unit is milliseconds. By default, the client must send a message within this time, otherwise it is considered dead.

Hbase.regionserver.handler.count

Default: 10

The number of RPC Server instances offered by RegionServers. For Master, this attribute is the number of handler offered by Master

Hbase.regionserver.msginterval

Default: 3000

Interval between RegionServer sending messages to Master (in milliseconds)

Hbase.regionserver.optionallogflushinterval

Default: 1000

The interval between synchronizing Hlog to HDFS. If the Hlog does not accumulate to a certain amount, synchronization will also be triggered when the time is up. The default is 1 second, in milliseconds.

Hbase.regionserver.regionSplitLimit

Default: 2147483647

When the number of region reaches this value, it will not split again. This is not a hard limit on the number of region. But it plays a guiding role. The default is MAX_INT (unlimited).

Hbase.regionserver.logroll.period

Default: 3600000

The interval at which the commit log is submitted, regardless of whether enough values have been written.

Hbase.regionserver.hlog.enabled

Default: true

Whether to enable WAL

Hbase.regionserver.wal.enablecompression

Default: false

Whether to compress the log

Hbase.regionserver.hlog.reader.impl

Default: org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader

The realization of HLog file reader.

Hbase.regionserver.hlog.writer.impl

Default: org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter

The realization of HLog file writer.

Hbase.regionserver.thread.splitcompactcheckfrequency

Default: 20000

How often does region server perform split/compaction checks?

Hbase.regionserver.nbreservationblocks

Default: 4

The number of block stored in memory. When an out of memory exception occurs, we can use these memory RegionServer to clean up before stopping.

Hbase.regionserver.dns.interface

Default: default

When using DNS, the network interface name of the IP address that RegionServer uses to report.

Hbase.regionserver.dns.nameserver

Default: default

When using DNS, RegionServer uses the domain name or IP address of DNS, which RegionServer uses to determine the domain name used to communicate with master.

Hbase.regions.slop

Default: 0

When any regionserver has average + (average * slop) region, it will execute Rebalance.

Hbase.regionserver.global.memstore.upperLimit

Default: 0.4

The maximum of all memtores for a single region server. Beyond this value, a new update operation is suspended, forcing the flush operation.

Hbase.regionserver.global.memstore.lowerLimit

Default: 0.35

When the flush operation is enforced, flush stops when it falls below this value. The default is 35% of the heap size. If this value is the same as hbase.regionserver.global.memstore.upperLimit, it means that when the update operation is suspended due to memory constraints, the flush will be executed as little as possible.

Hbase.coprocessor.regionserver.classes

The name of the coprocessor class used by Master. Use comma separation in the case of multiple classes. For example, the use of org.apache.hadoop.hbase.security.access.AccessController provides security control capabilities.

Default: null.

3. Client related parameters

Hbase.client.write.buffer

Default: 2097152

The default size of the write buffer for HTable clients. Because buffering requires application space on the client and server, it consumes memory in both the client and the server. Larger buffer can reduce the number of RPC, thereby improving performance. Estimate the memory occupied by the server: hbase.client.write.buffer * hbase.regionserver.handler.count

Hbase.client.pause

Default: 1000

The pause time of the retry after a failed client operation.

Hbase.client.retries.number

Default: 10

The maximum number of retries after a client failure. For example, region query, get,Update and other operations.

Hbase.client.scanner.caching

Default: 1

The number of rows fetched from the server at a time when the next method of Scanner is called and the value is not in the cache. A higher value means that Scanner is faster, but takes up more memory. When the buffer is full, the next method call becomes slower and slower. Slow to a certain extent may lead to a timeout. For example, it exceeds the hbase.regionserver.lease.period.

Hbase.client.keyvalue.maxsize

Default: 10485760

The maximum size of a KeyValue instance. This is used to set the upper bound of the size of a single entry in the storage file. Because a KeyValue cannot be split, it can be avoided that the region is indivisible because the data is too large. It would be wise to set it to the number that is divisible by the maximum region size. If set to 0 or less, this check is disabled. Default 10MB.

4. ZooKeeper correlation

Hbase.zookeeper.dns.interface

When using DNS, the network interface name of the IP address that Zookeeper uses to report.

Default: default

Hbase.zookeeper.dns.nameserver

Default: default

When using DNS, Zookeepr uses the domain name or IP address of DNS, which Zookeeper uses to determine the domain name used to communicate with master.

Zookeeper.session.timeout

Default: 180000

ZooKeeper session timeout. HBASE passes this value to the zk cluster, recommending the maximum timeout for a session. The unit is milliseconds.

Zookeeper.znode.parent

Default: / hbase

The root ZNode of the HBase in ZooKeeper. All HBase ZooKeeper will use this directory to configure relative paths. By default, all HBase ZooKeeper file paths are relative, so they all go to this directory.

Zookeeper.znode.rootserver

Default: root-region-server

The path to the root region saved by ZNode. This value is written by Master and read by client and regionserver. If set to a relative address, the parent directory is ${zookeeper.znode.parent}. By default, it means that the path to the root region is stored in / hbase/root-region-server.

Hbase.zookeeper.quorum

Default: localhost

The address list of the Zookeeper cluster, separated by commas. For example: host1.mydomain.com,host2.mydomain.com,host3.mydomain.com. The default is localhost, and if HBASE_MANAGES_ZK is set in hbase-env.sh, these ZooKeeper nodes will be started with HBase.

Hbase.zookeeper.peerport

Default: 2888

The port used by the ZooKeeper node.

Hbase.zookeeper.leaderport

Default: 3888

ZooKeeper is used to select the port of the Leader.

Hbase.zookeeper.property.initLimit

Default: 10

Configuration in zoo.conf of ZooKeeper. Limit the number of ticks in the initialization synchronization phase

Hbase.zookeeper.property.syncLimit

Default: 5

Configuration in zoo.conf of ZooKeeper. The limit on the number of ticks between sending a request and gaining recognition

Hbase.zookeeper.property.dataDir

Default: ${hbase.tmp.dir} / zookeeper

Configuration in zoo.conf of ZooKeeper. Storage location of the snapshot

Hbase.zookeeper.property.clientPort

Default: 2181

Configuration in zoo.conf of ZooKeeper. The port to which the client connects

Hbase.zookeeper.property.maxClientCnxns

Default: 2000

Configuration in zoo.conf of ZooKeeper. The number of concurrent requests for a single Client (distinguished by IP) accepted by a single node in a ZooKeeper cluster. This value can be adjusted a little higher to prevent problems in stand-alone and pseudo-distributed modes.

5. Patterns are related to storage

Hbase.table.max.rowsize

Default: true

When set to true, the tables in zookeeper are locked when the schema is in reverse change operations to avoid breaking table consistency in parallel operations.

Hbase.table.max.rowsize

Default: 1073741824

The byte size of the maximum row obtained when the scan option in the Get and Scan operations is not specified. If it is exceeded, a RowTooBigException exception is thrown.

Hbase.hregion.memstore.flush.size

Default: 67108864

When the size of the memstore exceeds this value, it will flush to disk. This value is checked by a thread every other hbase.server.thread.wakefrequency.

Hbase.server.thread.wakefrequency

Default: 10000

The sleep interval in milliseconds for Service threads (log roller, etc.).

Hbase.hregion.preclose.flush.size

Default: 5242880

When the size of the memstore in a region is greater than this value, we trigger the close. It will first run the "pre-flush" operation, clean up the memstore that needs to be closed, and then take the region offline. When a region goes offline, we can't do any more writes. If a memstore is very large, the flush operation will take a lot of time. The "pre-flush" operation means that the memstore is emptied before the region goes offline. In this way, when the close operation is finally performed, the flush operation will be very fast.

Hbase.hregion.memstore.block.multiplier

Default: 2

If memstore has the size of a hbase.hregion.flush.size that is a multiple of hbase.hregion.memstore.block.multiplier, the update operation is blocked. This is to prevent the loss of control caused by the peak of update. If there is no upper bound, flush will take a long time to merge or split, and in the worst case, an out of memory exception will be thrown.

Hbase.hregion.memstore.mslab.enabled

Default: false

Experience feature: enable memStore to allocate local buffers. The purpose of this feature is to prevent excessive fragmentation of the heap when there is a large write load. This reduces the frequency of GC operations.

Hbase.hregion.max.filesize

Default: 268435456

Maximum HStoreFile size. If the HStoreFile growth of a Column families reaches this value, the Hegion will be cut into two. Default: 256M.

Hbase.hstore.compactionThreshold

Default: 3

When a HStore contains more than this value of HStoreFiles, a merge operation is performed to write the HStoreFiles into one. The higher this value, the longer it takes to merge.

Hbase.hstore.blockingStoreFiles

Default: 7

When a HStore contains more than this value of HStoreFiles, a merge operation is performed, and the update blocks until the merge is complete, until the value of hbase.hstore.blockingWaitTime is exceeded

Hbase.hstore.blockingWaitTime

Default: 90000

The number of StoreFile limited by hbase.hstore.blockingStoreFiles will cause update blocking, which is used to limit the blocking time. When this time is exceeded, HRegion will stop blocking update operations, but the merge has not yet been completed. The default is 90s.

Hbase.hstore.compaction.max

Default: 10

The maximum number of HStoreFiles merged per "small".

Hbase.hregion.majorcompaction

Default: 86400000

The interval between the major compactions of all HStoreFile in a Region. The default is 1 day. Setting to 0 disables this feature.

Hbase.mapreduce.hfileoutputformat.blocksize

Default: 65536

HFileOutputFormat in MapReduce can write storefiles/hfiles. This value is the minimum blocksize for hfile. Usually when HBase writes Hfile, bloocksize is determined by table schema (HColumnDescriptor), but when mapreduce writes, we cannot get the blocksize in schema. The smaller the value, the larger your index, and the smaller the data you need to fetch for random access. If your cell is small and you need faster random access, you can lower this value.

Hfile.block.cache.size

Default: 0.2

The percentage of block cache allocated to HFile/StoreFile to the largest heap (- Xmx setting). The default is 20%. Setting it to 0 means no allocation.

Hbase.hash.type

Default: murmur

The hash algorithm used by the hash function. You can use MurmurHash and jenkins (JenkinsHash). Provided to bloom filters for use.

6. Safety related

Hbase.security.authentication

The permission control mode of HBase. Optional inputs are simple and kerberos.

Default value: simple

Hbase.security.authentication

Whether to enable HBase security authentication.

Default: false

Hbase.superuser

Super Admin user for HBase.

Default value: hbase

Hbase.master.keytab.file

Default:

Open kerberos-based Security. HMaster Server verifies the path of the keytab file used for login.

Hbase.master.kerberos.principal

Default:

Open kerberos-based Security. HMaster uses principal name.

Hbase.regionserver.keytab.file

Default:

Open kerberos-based Security. HRegionServer verifies the path of the keytab file used for login.

Hbase.regionserver.kerberos.principal

Default:

Open kerberos-based Security. The principal name used by RegionServer.

The above is all the contents of the article "what are the common parameters of HBase". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.