Shulou(Shulou.com)06/02 Report--

What is the internal mechanism of Hbase? I believe many inexperienced people are at a loss about it. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

As we all know, HBase is a non-relational database, which is characterized by distribution, column storage, open source and versioning. Whether in big data's interview or big data's work, this is a common problem, but few people can explain the internal mechanism of Hbase. Today we will take a moment to talk about the internal mechanism of Hbase.

In HBase, whether you add new lines or modify existing lines, the internal process is the same. After receiving the command, HBase saves the change information, or fails to write and throws an exception. By default, when a write is performed, it is written to two places: write-ahead log (also known as HLog) and MemStore. The default way for HBase is to record writes in these two places to ensure data persistence. The write action is considered complete only when the change information in these two places is written and confirmed.

MemStore is a write buffer in memory where data in HBase accumulates before it is permanently written to the hard disk. When the MemStore fills up, the data is written to the hard disk and a HFile is generated. HFile is the underlying storage format used by HBase. HFile corresponds to column families. A column family can have more than one HFile, but one HFile cannot store data for multiple column families. On each node of the cluster, each column family has a MemStore.

Hardware failures are common in large distributed systems, and HBase is no exception. Imagine that if the MemStore is not written, the server crashes and data that is not written to the hard disk in memory will be lost. The response for HBase is to write to WAL before the write action is completed. Each server in the HBase cluster maintains a WAL to record changes. WAL is a file on the underlying file system. The write action is not considered to have completed successfully until the new WAL record is successfully written. This ensures that HBase and the file system that supports it are persistent. In most cases, HBase uses the Hadoop distributed File system (HDFS) as the underlying file system.

If the HBase server goes down, data that is not written from MemStore to HFile can be recovered by playing back WAL. You don't have to do it by hand. There is a recovery process part of the internal mechanism of Hbase. Each HBase server has a WAL, and all tables on this server (and their column families) share this WAL.

You might think that skipping WAL while writing should improve write performance. However, we do not recommend disabling WAL unless you are willing to lose data if something goes wrong. If you want to test it, the following code can disable WAL: note: not writing WAL increases the risk of data loss in the event of a RegionServer failure. If WAL is turned off, HBase may not be able to recover data in the event of a failure, and all written data not written to the hard disk will be lost.

After reading the above, have you mastered the internal mechanism of Hbase? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.