Shulou(Shulou.com)06/03 Report--

Premise: jdk, non-secret interworking, time synchronization, zookeeper,hadoop

Zookeeper document

Hadoop document

Time server building

Cluster

Hostname studycentos156clientIP192.168.233.155192.168.233.156192.168.233.158 Service 1zookeeper 1zookeeper 2zookeeper 3 Service 2namenodenamenode Service 3datanodedatanodedatanode Service 4journalnodejournalnode Service 5nodemanagernodemanagernodemanager Service 6zkfczkfc Service 7ResourceManager Service 8hmasterhmasterhmaster Service 9ResourceManager Node 1-study

# download hbase: official download path

# version 1.2.6 for this document

Cd / tmp/

Wget https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/1.2.6/hbase-1.2.6-bin.tar.gz

# decompression

Tar axf hbase-1.2.6-bin.tar.gz-C / usr/local/

# rename

Mv / usr/local/hbase-1.2.6/ / usr/local/hbase

# modify environment variables

Vim / etc/profile

# hbase

Export HBASE_HOME=/usr/local/hbase

Export PATH=$PATH:$HBASE_HOME/bin

Source / etc/profile

# modify configuration file

Cd $HBASE_HOME/conf

Vim hbase-env.sh

# add as below

Export JAVA_HOME=/usr/local/jdk

Export HBASE_MANAGES_ZK=false

To modify JVM, you only need to modify HBASE_MASTER_OPTS and HBASE_REGIONSERVER_OPTS in hbase-env.sh.

Vim hbase-site.xml

Hbase.rootdir

Hdfs://hadoop/hbase

The shared directory of region server, which specifies the directory in hdfs (if hdfs does HA, it should point to the logical name of hdfs in core-site.xml, and hbase will fail if the state transition occurs in namenode. / hbase is specified here, and it can be written in the local directory. Default is file:///tmp/hbase-${user.name}/hbase.

Hbase.cluster.distributed

True

The operation mode of HBase, false is stand-alone mode, true is distributed mode

Hbase.zookeeper.property.clientPort

2181

Zookeeper connection port

Hbase.zookeeper.quorum

Study,centos156,client

Zookeeper node address

Zookeeper.session.timeout

60000

Session timeout in milliseconds

Hbase.zookeeper.property.dataDir

/ usr/local/zookeeper

Configuration in zoo.conf of ZooKeeper, snapshot storage location

Hbase-site.xml configuration parameters reference documentation

# specify regionservers node

Vim regionservers

Study

Centos156

Client

# copy files to other nodes

Scp-r / usr/local/hbase/ centos156:/usr/local/

Scp-r / usr/local/hbase/ client:/usr/local/

Scp / etc/profile centos156:/etc/

Scp / etc/profile client:/etc/

Node 2-centos156

Source / etc/profile

Node 3-client

Source / etc/profile

Start cluster node 1-study

Start-hbase.sh

The warning message here is because I use the jdk of 8.

It doesn't matter.

View statu

Hbase shell

# status View status

Start hmaster on other machines as a backup node

Node 2-centos156

Hbase-daemon.sh start master

Node 3-client

Hbase-daemon.sh start master

View statu

Hbase shell

# status View status

Turn off the hmaster test on add one

Hbase-daemon.sh stop master

Hbase data migration

Now look where you put the data in the hbase-site.xml file.

Hbase.rootdir hdfs://study:9000/hbase

Because we put it in hadoop, we only need to migrate the data of hadoop.

Data backup

Mkdir / tmp/hbase

Chmod 777 / tmp/hbase

Hadoop fs-copyToLocal hdfs://study:9000/ / tmp/hbase

Data recovery

First transfer the file to the target machine using a U disk or any other means

Hadoop fs-copyFromLocal / tmp/hbase hdfs://study:9000/

Hadoop fs-ls /

Restart hbase

Stop-hbase.sh

Start-hbase.sh

Global modification of hbase JVM settings

-Xmx Heap (heap memory) maximum allowable value

-Xms Heap (heap memory) minimum allowable value

-Xmn younger generation size

Vim hbase-env.sh

Export HBASE_OPTS= "- XX:+UseConcMarkSweepGC-Xmx4000m-Xms4000m-Xmn1000m-XX:+UseParNewGC-XX:CMSInitiatingOccupancyFraction=70" master modification

Vim hbase-env.sh

Export HBASE_MASTER_OPTS= "$HBASE_MASTER_OPTS $HBASE_JMX_BASE-Xmx4000m-Xms4000m-Xmn1000m-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=70" regionserver modification

Vim hbase-env.sh

Export HBASE_REGIONSERVER_OPTS= "$HBASE_REGIONSERVER_OPTS $HBASE_JMX_BASE-Xmx4000m-Xms4000m-Xmn1000m-XX:+UseParNewGC-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=70" hbase exception handling

Test hive and spark to rebuild the hadoop cluster, and the following error is encountered

2018-01-10 11 master.HMasterCommandLine 21 ERROR [main] master.HMasterCommandLine: Master exitingjava.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster. At org.apache.hadoop.hbase.master.HMaster.constructMaster (HMaster.java:2512) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster (HMasterCommandLine.java:231) at org.apache.hadoop.hbase.master.HMasterCommandLine.run (HMasterCommandLine.java:137) at org.apache.hadoop.util.ToolRunner.run (ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain (ServerCommandLine.java:126) At org.apache.hadoop.hbase.master.HMaster.main (HMaster.java:2522) Caused by: java.io.IOException: Port 9000 specified in URI hdfs://master:9000/hbase but host 'master' is a logical (HA) namenode and does not use port information. At org.apache.hadoop.hdfs.NameNodeProxies.createFailoverProxyProvider (NameNodeProxies.java:488) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy (NameNodeProxies.java:148)

Recall the previously built process (flipping documents. ) found that the time is not synchronized

After the time synchronization is done, the startup still reports an error like this. Suspected of being caused by the cache, he deleted all the files under / tmp and hbae angrily, and then rebuilt the hbase.

And then the problem was solved.

From the point of view of the phenomenon, the main reason for the problem is that the time between servers is not synchronized.

HMaster can't get up.

Check the error report

Org.apache.hadoop.ipc.RemoteExceptionOperation category READ is not supported in state standby

This is an exception in the hadoop cluster. Check the cluster status and check the hadoop documentation for details.

Error

2018-01-22 20 master.HMaster 58 FATAL [slave2:16000.activeMasterManager] master.HMaster: Failed to become active masterorg.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode= "/ hbase/.tmp": hadoop:supergroup:drwxr-xr-x

This is because the root user does not have write permission to the directory in hadoop and starts or returns the hadoop directory permission with the previously used user.

Hbase shell common commands

Hbase shell reference documentation

Help 'get' view help information

Status View status

List View list

Scan looks at all the data in a table

Scan, view all the data in a column of the table

Create, ["column name 2"] create a table

Describe view table structure

Is_enabled determines whether the table is enabled or not

Get, get all the data of a key

Get, get all the data in a column cluster of a key

Filter modifier for filtering information (see reference documentation for details)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.