Shulou(Shulou.com)06/01 Report--

This article mainly introduces "problems encountered in the process of HBase migration and solutions". In daily operation, I believe that many people have doubts about problems and solutions encountered in the process of HBase migration. Xiaobian consulted all kinds of materials and sorted out simple and easy-to-use methods of operation. I hope it will be helpful to answer the doubts of "problems encountered in the process of HBase migration and solutions". Next, please follow the editor to study!

one。 Problems encountered in the migration process and solutions

Customer HBase version: Version 0.94.15

Tencent big data Suite HBase version: Version 1.2.1

Customer private cloud system version (test): tlinux1.2

The problems encountered and the resolution process are as follows:

Abnormal phenomena of 1.HBase operation (date and hwclock)

The operation of HBase is occasionally abnormal, and the components stop running. See the log for information such as time differences, but the date view is completely consistent, thinking that it may be the difference in hardware time. Through hwclock, there is indeed a great difference, and it is basically restored after adjustment through hwclock-w. It is confirmed that hardware time synchronization is only performed on machines in Tencent Cloud environment in the initialization script, which has been optimized.

Abnormal operation of 2.HBase II (hostname and / etc/resolv.conf)

HBase is not working properly again, and the component stops running. Look at the following errors through the log

ERROR [regionserver//10.0.0.106:16020] regionserver.HRegionServer: Master passed us a different hostname to use; was=10.0.0.106, but now=host-10-0-0-106.openstacklocal

Through hostname, we can see that all machine hostname are private network IP. We guess that the inconsistency may be caused by querying which table during network interaction. Check the dns parsing information as follows

[root@10 ~] # hostname10.0.0.106; generated by / sbin/dhclient-script#search openstacklocal 0.0.106#nameserver 10.0.0.2#nameserver 10.0.0.3

In the case of search openstacklocal, the guess is the abnormal behavior of the virtual machine. Comment out the relevant search information in resolv.conf, stop the nscd service, and restart HBase. This error does not occur again, and the HBase is running normally.

3. The discovery and repair process of snappy needs to be supported:

The official import/export tool is planned to be used in the process of table migration. The first step is to create a table in the target cluster. After the table is created in the target cluster through desc information, list can see the table. After querying through scan, the table cannot be queried. There are the following errors in the log:

Org.apache.hadoop.HBase.DoNotRetryIOException: Compression algorithm 'snappy' previously failed test.

Through google query, HBase is required to support snappy compression algorithm. Through hadoop checknative, it is found that the cluster does not support snappy algorithm by default (although snappyrpm is installed)

Native library checking:hadoop: true / data/tbds-base/usr/hdp/2.2.0.0-2041/hadoop/lib/native/libhadoop.sozlib: true / lib64/libz.so.1snappy: falselz4: true revision:99bzip2: falseopenssl: false build does not support openssl.

Through the method of manually building a table, you can view the table information by list after building a table with the following desc information. Scan was unable to view the table contents, and the log found the following error

Desc Information:

COLUMN FAMILIES DESCRIPTION {NAME = > 'Aids, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 013, VERSIONS = >'1', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 02, TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true' METADATA = > {'ENCODE_ON_DISK' = >' true'}} {NAME = > 'Dache, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 0mm, VERSIONS = > '2147483647', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 01', TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true', ENCODE_ON_DISK = >' true'}

Error message:

Org.apache.hadoop.HBase.DoNotRetryIOException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support

You can add the attribute HBase.regionserver.codecs value to snappy in HBase-site.xml. After passing this method in the test cluster, HBase failed to start.

Confirm that snappy is supported on tlinux1.2 's hadoop cluster: that is, you need to compile hadoop-related local libraries (native libraries) on a specific system to replace hadoop's current native libraries, and then add the hadoop home directory to the startup environment script of HBase.

At present, the nativesnappy library of hadoop under tlinux1.2 is used on the current network. At the same time, you need to ensure that the library of this hadoop can be referenced to libjvm.so (a so file of jre) to directly replace the native directory under hadoop/lib, ensure that the rpm package of snappy has been installed, and add HADOOP_HOME= {Hadoop installation home directory} to HBase-env.sh. After re-hadoop checknative, it is found that snappy is supported. Restart HBase step by step.

Native library checking:hadoop: true / data/tbds-base/usr/hdp/2.2.0.0-2041/hadoop/lib/native/libhadoop.sozlib: true / lib64/libz.so.1snappy: true / usr/lib64/libsnappy.so.1lz4: true revision:99bzip2: falseopenssl: false build does not support openssl.

Migration method from 4.HBase0.9.4 Cluster data Table to HBase1.2.1 Cluster data Table

Violent migration reference http://my.oschina.net/CainGao/blog/616502

1) find the directory location of the source table on the hdfs, and move the directory directly to the table root directory of the target cluster HBase table on the target cluster hdfs.

2) when migrating violently, the tableinfo information is a file, namely .tableinfo.00000001. Version 0.9.4 this file is located in the root directory of the HBase table on hdfs, while the 1.2.1 file is located in the root directory of the HBase table directory on hdfs. / tabledesc directory, you need to manually create this directory and adjust the location of this file

3) modify the owner information of the copied table catalog file

4) restart all components of HBase

5) at this point, you can log in to HBaseshell to view the migrated table through list, but operations such as scan will fail

6) repair the meta information through HBase hbck-fixMeta; repair the partition by HBase hbck-fixAssignments. During the operation of these two steps, pay attention to observe whether the log is abnormal. There are a large number of errors when you try this method for the first time in practice. It is found that the error content is related to snappy. After snappy is supported, the table information is checked and the table content is normal. The comparison of randomly selected table content is also normal. It can be considered that this method has been migrated successfully.

7) when migrating through the import/export method, you need to manually create a target table in the target cluster. The table structure of the source cluster is as follows:

Import/export reference address

COLUMN FAMILIES DESCRIPTION {NAME = > 'Aids, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 013, VERSIONS = >'1', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 02, TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true' METADATA = > {'ENCODE_ON_DISK' = >' true'}} {NAME = > 'Dache, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 0mm, VERSIONS = > '2147483647', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 01', TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true', ENCODE_ON_DISK = >' true'}

The following error occurred while creating a new table from this desc information:

Unknown argument ignored for column family A: ENCODE_ON_DISK

Manual test as long as add this parameter ENCODE_ON_DISK to create the table will certainly occur this error, the table will be successful, but there is no such field in the table information. After checking the code of look, it is found that this field is obsolete in the new version, but the old cluster of customers needs this field. After it cannot be written normally by import method and is successfully migrated through step 6), the desc information of the table is as follows:

COLUMN FAMILIES DESCRIPTION {NAME = > 'Aids, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 013, VERSIONS = >'1', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 02, TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true' METADATA = > {'ENCODE_ON_DISK' = >' true'}} {NAME = > 'Dache, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 0mm, VERSIONS = > '2147483647', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 01', TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true', METADATA = > {' ENCODE_ON_DISK' = > 'true'}'

Old cluster table structure

COLUMN FAMILIES DESCRIPTION {NAME = > 'Aids, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 013, VERSIONS = >'1', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 02, TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true' METADATA = > {'ENCODE_ON_DISK' = >' true'}} {NAME = > 'Dache, DATA_BLOCK_ENCODING = >' NONE', BLOOMFILTER = > 'NONE', REPLICATION_SCOPE = >' 0mm, VERSIONS = > '2147483647', COMPRESSION = > 'SNAPPY', MIN_VERSIONS = >' 01', TTL = > 'FOREVER', KEEP_DELETED_CELLS = >' false', BLOCKSIZE = > '65536', IN_MEMORY = >' false', BLOCKCACHE = > 'true', ENCODE_ON_DISK = >' true'}

You can see that there is a difference in the definition of ENCODE_ON_DISK fields in the old and new versions, so we tested that the table was built with the above desc information in the new cluster, and then imported into HBase through the import method. As a result, there is still no data written, and it can be concluded that this parameter ENCODE_ON_DISK is completely obsolete in HBase1.2.1, and the new version uses an entire field to wrap this information. When the old cluster has parameters, the official import/export method is temporarily unavailable for direct migration from HBase0.9.8 to HBase1.2.1.

At this point, the study of "problems encountered in the process of HBase transfer and solutions" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.