Shulou(Shulou.com)06/01 Report--

This article is to share with you about the basic principles of HBase, the editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

Start with BigTable.

HBase is an open source implementation based on Google BigTable. It is a highly reliable, high-performance, column-oriented, scalable distributed database that can be used to store unstructured and semi-structured sparse data. HBase supports very large-scale data storage and can scale horizontally to handle data tables consisting of more than 1 billion rows of data and millions of columns of elements.

BigTable is a distributed storage system, which uses the MapReduce distributed parallel computing model proposed by Google to deal with massive data, uses Google distributed file system GFS as the underlying data storage, and uses Chubby to provide collaborative service management. It has the characteristics of extensive application, scalability, high availability and high performance. For a comparison between BigTable and HBase, see the following table:

MapReduce Collaborative Service ChubbyZookeeperCAP Theory based on BigTbaleHBase data Storage GFSHDFS data processing MapReduceHadoop

In 2000, a professor Eric Brewer of Berkerly University proposed a CAP theory. In 2002, Seth Gilbert (Seth Gilbert) and Nancy Lynch (Nancy Lynch) of the Massachusetts Institute of Technology published the proof of Brewer conjecture, which proved the correctness of CAP theory. The so-called CAP theory means that it is impossible for a distributed computing system to meet the following three points at the same time:

Consistency (Consistency)

It is equivalent to all nodes accessing the same latest copy of data. That is, any read operation can always read the results of previously completed write operations, that is, in a distributed environment, the data accessed by different nodes is consistent.

Availability (Availability)

You can get a non-incorrect response for each request-- but there is no guarantee that the data obtained will be up-to-date. That is, the data can be obtained quickly and the operation result can be returned within a certain period of time.

Partition fault tolerance (Partition tolerance)

In terms of practical effect, partitioning is equivalent to the time limit for communication. If the system cannot achieve data consistency within the time limit, it means that a partition situation has occurred and a choice must be made between C and A for the current operation. That is, when there is a network partition (some nodes in the system can not communicate with other nodes), the separate system can also operate normally, that is, reliability.

As shown in the figure above: a distributed system cannot satisfy both consistency, availability, and partition fault tolerance, at most two at the same time. When dealing with CAP issues, you have the following options:

It satisfies CA but does not satisfy P. Putting all the transaction-related content on the same machine will affect the scalability of the system. Traditional relational database. Such as MySQL, SQL Server, PostgresSQL and so on all adopt this kind of design principle.

Satisfy AP, but not C. Does not meet consistency (C), that is, allows the system to return inconsistent data. In fact, for WEB2.0 sites, it is more concerned about the availability of services than consistency. For example, if you post a blog or write a Weibo post, some of your friends see the article or Weibo immediately, while others have to wait a while before they can post it or Weibo. Although there is a delay, for an entertainment Web 2.0 site, the delay of a few minutes is not important and will not affect the user experience. On the contrary, when you post an article or Weibo, you can't post it immediately (usability is not satisfied), and users must be upset about it. So, for WEB2.0 sites, usability and partition fault tolerance take priority over data consistency, of course, not completely abandon consistency, but the ultimate consistency (with delay). This principle is adopted in NoSQL databases such as Dynamo, Cassandra, CouchDB and so on.

It satisfies CP, but not A. Emphasize consistency (C) and partition fault tolerance (P), and give up availability (A). When a network partition occurs, the affected service needs to wait for the data to be consistent and cannot provide services during the waiting period. Such as Neo4J, HBase, MongoDB, Redis, etc., adopt this kind of design principle.

Why NoSQL appears

The so-called NoSQL, the abbreviation of Not Only SQL, means more than just SQL. The CAP theory mentioned above is the design principle of NoSQL. So why the rise of NoSQL databases? Because of the arrival of WEB2.0 and big data era, relational database is more and more unable to meet the demand. With the development of big data, Internet of things, mobile Internet and cloud computing, the proportion of unstructured data is more than 90%. Relational database has exposed more and more defects in the face of big data because of its inflexible model and poor expansion level. As a result, NoSQL database came into being to better meet the needs of big data era and WEB2.0.

Faced with the challenges of WEB2.0 and big data, relational databases perform poorly in the following aspects:

Poor performance for massive data processing.

In the WEB2.0 era, especially with the development of the mobile Internet, UGC (user generated content, User Generated Content) and PGC (Public generated content, Public Generated Content) occupy our daily life. Nowadays, with the development of self-media everywhere, almost everyone has become the creator of content, such as blog posts, comments, opinions, news, videos and so on. It can be seen that the speed at which these data are generated is so fast and the amount of data is so large. For example, Weibo, official account, or Taobao, the data generated in one minute may be very amazing. In the face of these tens of millions of levels of data records, the query efficiency of relational database is obviously unacceptable.

Unable to meet high concurrency requirements

In the era of WEB1.0, most of them are static web pages (that is, what is provided), so that when large-scale users visit, they can achieve better response ability. However, in the era of WEB2.0, the emphasis is on user interaction (user-created content). All information needs to be generated dynamically, which will lead to high concurrency of database access and possibly tens of thousands of read and write requests per second, which is unbearable for many relational databases.

Unable to meet the needs of scalability and high availability

In today's era of entertainment to death, hot issues (attracting attention, satisfying novelty hunter) will attract a swarm of traffic, such as Weibo exposed a star cheating, the hot search list will quickly attract a large number of users (commonly known as onlookers), resulting in a large number of interactive exchanges (hot spots), which will lead to a sharp increase in the reading and writing load of the database. As a result, the database needs to be able to quickly improve its performance in a short period of time to cope with unexpected needs (after all, downtime opportunities have a great impact on the user experience). However, relational databases are usually difficult to scale horizontally, and can not simply expand performance and load capacity by adding more hardware and service nodes like web servers and application servers.

To sum up, NoSQL database came into being, which is the necessity of the development of IT.

The characteristics and usage scenarios of HBase strong consistency in reading and writing

HBase is not a final consistent (eventually consistent) data store. This makes it suitable for high-speed counting aggregation tasks.

Automatic slicing (Automatic sharding)

The HBase table is distributed in the cluster through region. As data grows, region automatically splits and redistributes

RegionServer automatic failover

Hadoop/HDFS integration

HBase supports external HDFS as its distributed file system.

MapReduce integration

HBase supports large concurrency processing through MapReduce, and HBase can be both source (Source) and sink (Sink)

Java client API

HBase supports easy-to-use Java API for programmatic access

Thrift/REST API

Access to HBase by supporting Thrift and REST

Block Cache and Bloom filter (Bloom Filter)

HBase supports Block Cache and Bloom filter for query optimization to improve query performance.

Operation and maintenance management

HBase provides built-in web pages and JMX metrics for operation and maintenance

Working with scen

HBase is not suitable for all scenarios

First of all, * * the amount of data * *. Make sure that there is enough data, if there are hundreds of millions or billions of rows of data, at least more than 10 million data in a single table, HBase would be a good choice. If there are only thousands or millions of lines, traditional RDBMS may be a better choice.

Secondly, the characteristics of relational database. Make sure you don't have to rely on all the extra features of RDBMS (such as column data types, secondary indexes, transactions, advanced query languages, etc.). An application based on RDBMS can not be migrated to HBase simply by changing the JDBC driver. It needs a complete redesign.

Thirdly, the hardware aspect. Make sure you have enough hardware. For example, because the default copy of HDFS is 3, it takes at least five data nodes to give full play to its characteristics, plus a NameNode node.

Finally, data analysis. Data analysis is a weakness of HBase, because table associations are basically not supported for HBase and even the entire NoSQL ecosystem. If the main requirement is data analysis, such as reporting, it is obvious that HBase is not appropriate.

Basic terminology of data model for HBase

HBase is a sparse, multi-dimensional, persistent storage mapping table, indexed by row key, column family, and column qualification in accordance with the timestamp, and the value of each cell is a byte array byte []. To understand HBase, you need to know the following concepts:

Namespace

Namespace, or namespaces, is a logical grouping of tables, similar to the database of a relational database management system. HBase has two predefined special namespaces: hbase and default, where hbase belongs to the system namespace and is used to store HBase's internal tables. Default belongs to the default namespace, that is, if no namespace is specified when the table is created, default is used by default.

Table

Consists of rows and columns, which are divided into several column families

OK

Row key is an uninterpreted array of bytes, and within HBase, row key is stored in the table from lowest to highest in dictionary order. The table of each HBase consists of several rows, each identified by a row key (row key). You can take advantage of this feature to store rows that are often read together.

Column family

In HBase, columns are organized by column families. All column members of a column family have the same prefix, for example, columns courses:history and courses:math are members of the column family courses. The colon (:) is the delimiter of the column family and is used to distinguish between prefixes and column names. Column families must be declared when the table is created, and columns can be declared when they are used. In addition, all data stored in a column family usually has the same data type, which can greatly increase the compression ratio of the data. Physically, members of a column family are stored together on the file system.

Column

The data in the column family is located by the column qualifier. Columns usually do not need to be defined when the table is created, nor do they need to be consistent across rows. The column has no explicit data type and is always treated as a byte array byte [].

Cell

Cells, that is, the specific stored data determined by row key, column families, and columns. The data stored in the cell also has no explicit data type and is always regarded as a byte array byte []. In addition, the data in each cell is multi-version, and each version corresponds to a timestamp.

Time stamp

Because HBase's table data is versioned, these versions are identified by timestamps. Each time a cell is modified or deleted, HBase automatically generates and stores a timestamp for it. Different versions of a cell are stored according to the descending order of timestamps, that is, priority is given to reading the latest data.

For more information about the data model of HBase, see the following figure:

Conceptual model

In the HBase conceptual model, a table can be thought of as a sparse, multidimensional mapping, as shown in the following figure:

As shown in the table above:

The table contains two rows of data, com.cnn.www and com.example.www

There are three column families: contents, anchor and people.

For the first row of data (the corresponding row key is com.cnn.www), the column family anchor contains two columns: anchor:cssnsi.com and anchor:my.look.ca; column family contents contains one column: contents:html

For the first row of data (the corresponding row key is com.cnn.www), contains 5 versions of the data

For the second row of data (the corresponding row key is com.example.www), contains 1 version of the data

In the above table, you can locate cell data through a four-dimensional coordinate: [row key, column family, column, timestamp], such as [com.cnn.www,contents,contents:html,t6]

Physical model

In terms of the conceptual model, HBase's tables are sparse. In physical storage, it is stored according to the column family. A column qualifier (column_family:column_qualifier) can be added to an existing column family at any time.

From the perspective of the physical model, the empty cells that exist in the conceptual model will not be stored. For example, if you want to access contents:html with a timestamp of T8, no value will be returned. It is worth noting that if no timestamp is specified when accessing the data, the latest version of the data is accessed by default because the data is sorted in descending order by version timestamp.

As shown in the above table: if you access row com.cnn.www and column contents:html, the data corresponding to T6 is returned if no timestamp is specified; similarly, if you access anchor:cnnsi.com, the data corresponding to T9 is returned.

The overall structure of the principle and operating mechanism of HBase

From the above description, you should have some understanding of HBase. Now let's take a look at the macro architecture of HBase, as shown in the following figure:

Let's first look at the overall architecture of HBase from a macro point of view. In terms of HBase deployment architecture, HBase has two types of servers: Master server and RegionServer server. Generally, a HBase cluster has one Master server and several RegionServer servers.

The Master server is responsible for maintaining the table structure information, and the actual data is stored on the RegionServer server. In the HBase cluster, the client gets the data from the client directly connected to the RegionServer, so you will find that you can still query the data after the Master is dead, but you cannot create a new table.

Master

As we all know, the master-slave architecture is adopted in Hadoop, that is, the namenode node is the master node and the datanode node is the slave node. The namenode node is critical to the hadoop cluster, and if the namenode node dies, the entire cluster is paralyzed.

However, in HBase clusters, the role of Master services is not that important. Although it is a Master node, it is not really a leader role. The Master service is more like a "busboy", similar to the role of a helper. Because when we connect to the HBase cluster, the client will get the address of the RegionServer directly from the Zookeeper and then get the desired data from the RegionServer without going through the Master node. In addition, when we insert data into the HBase table, delete data and other operations, we also interact directly with RegionServer, without the participation of Master services.

So what is the purpose of Master services? Master is only responsible for various coordination tasks, such as creating tables, deleting tables, moving Region, merging, and so on. One common problem with these operations is that they need to cross the RegionServer. Therefore, HBase assigns this work to the Master service. The advantage of this structure is that the cluster's dependence on Master is greatly reduced. Generally speaking, there are only one or two Master nodes. Once there is a downtime, if the cluster is highly dependent on Master, then a single point of failure will occur. In HBase, even if the Master goes down, the cluster can still run normally, and the data can still be stored and deleted.

RegionServer

RegionServer is the container where Region is stored. Intuitively speaking, it is a service on the server. RegionServer is the node that actually stores data, which is eventually stored in the distributed file system HDFS. When the client gets the address of the RegionServer from the ZooKeeper, it gets the data directly from the RegionServer. For HBase clusters, it is more important than Master services.

Zookeeper

RegionServer relies heavily on ZooKeeper services, and ZooKeeper plays a role similar to that of a butler in HBase. ZooKeeper manages all the RegionServer information of the HBase, including the RegionServer on which the specific data segments are stored. Every time the client connects to the HBase, it communicates with the ZooKeeper to find out which RegionServer needs to connect, and then connects to the RegionServer.

We can access the data of the hbase node through zkCli, and the information of the hbase:meta table can be obtained by the following name:

[zk: localhost:2181 (CONNECTED) 17] get / hbase/meta-region-server

A brief summary of the role of Zookeeper in the HBase cluster is as follows: for the server, it is an important dependence to achieve cluster coordination and control. For the client, it is an essential part of querying and manipulating data.

It should be noted that when the Master service is hung up, read and write operations can still be performed, but once the ZooKeeper is hung up, the data cannot be read, because the location of the metadata table hbase:meata needed to read the data is stored on the ZooKeeper. It can be seen that zookeeper is crucial to HBase.

Region

A Region is a collection of data. Tables in HBase typically have one or more Region. Region cannot cross servers. There are one or more Region on a RegionServer. When you start to create a table, when the amount of data is small, one Region is enough to store all the data. When the amount of data increases gradually, it will be split into multiple region;. When the HBase is performing load balancing, it is also possible to move Region from one RegionServer to another RegionServer. Region is stored in HDFS, and all its data access operations are implemented by calling the client interface of HDFS. A Region is equivalent to a partition of a partitioned table in a relational database.

Micro structure

The previous section explains the overall architecture of HBase, and then look at the internal details, as shown in the following figure: shows the internal architecture of a RegionServer.

As shown in the figure above: a RegionServer can store multiple region,Region equivalent to one data shard. Each Region has a starting rowkey and an ending rowkey, representing the range of rowkeys it stores. Within a region, it includes multiple store, one store corresponds to a column family, and each store contains a MemStore, which is mainly responsible for data sorting. After exceeding a certain threshold, the MemStore data is brushed to the HFile file, and the HFile file is the place where the data is finally stored.

It is worth noting that a RegionServer shares a WAL (Write-Ahead Log) pre-write log. If WAL is enabled, it will be written into WAL when writing data, which can play a fault-tolerant role. WAL is an insurance mechanism in which data is written to WAL before it is written to Memstore. In this way, the data can be recovered from the WAL when the fault is recovered. In addition, each Store has a MemStore for data sorting. A RegionServer also has only one BlockCache, which is used to read data for caching.

WAL pre-write log

* * Write Ahead Log (WAL) * * records all data in HBase. WAL plays the role of fault-tolerant recovery and is not a required option. On HDFS, the default path for WAL is / hbase/WALs/, users that can be configured through hbase.wal.dir.

WAL is on by default. If it is closed, you can use the following command Mutation.setDurability (Durability.SKIP_WAL). WAL supports both asynchronous and synchronous writes, which are done asynchronously by calling the following method Mutation.setDurability (Durability.ASYNC_WAL). Synchronization is done by calling the following method: Mutation.setDurability (Durability.SYNC_WAL), where synchronization is the default.

With regard to asynchronous WAL, synchronous data is not immediately triggered when there are Put, Delete, or Append operations. Instead, you have to wait for a certain time interval, which can be set by the parameter hbase.regionserver.optionallogflushinterval, which defaults to 1000ms.

MemStore

There is one instance of MemStore in each Store. After the data is written to WAL, it is put into MemStore. MemStore is a memory storage object, and only when the MemStore is full will the data be flush into the HFile.

In order to make the data stored sequentially and thus improve the reading efficiency, HBase uses the LSM tree structure to store the data. The data is first sorted into a LSM tree in Memstore, and then brushed to HFile.

About MemStore, it's easy to get confused. The data is already stored on HDFS's WAL before it is brushed to HFile, so why put it in MemStore? In fact, it is very simple, we all know that HDFS can not be modified, and HBase data is sorted by Row Key, in fact, the sorting process is carried out in MemStore. It is worth noting that the purpose of MemStore is not to speed up writing, but to sort Row Key.

HFile

HFile is the actual carrier of data storage, and all the tables, columns and other data we create are stored in HFile. When the Memstore reaches a certain threshold, or when the interval between brushing and writing is reached, the HBaes will be brushed to the HDFS system by the contents of the Memstore, which is called an HFile file stored on the hard disk. At this point, our data is really persisted to the hard disk.

The location of Region

Before we start to explain the data read and write process of HBase, let's take a look at how Region is positioned. We know that Region is a very important concept of HBase, and Region is stored in RegionServer, so how does the client locate the required region when reading and writing? On this issue, the old version of HBase is different from the new version of HBase.

Older version of HBase (before 0.96.0)

The older version of HBase uses a three-tier query architecture, as shown in the following figure:

Understand the basic principles of HBase / old positioning. PNG

As shown in the figure above: the first layer of positioning is the node data in Zookeeper, which records the location information of-ROOT- table.

The second layer-ROOT- table records the location information of .META.region. The-ROOT- table has only one region, and the .meta can be accessed through the-ROOT- table. Data in the table

The third layer. Meta. Table, which records the region location information of the user data table,. META. Tables can have more than one region.

The whole query steps are as follows:

Step 1: the user knows the RegionServer location of the-ROOT- table by looking for the / hbase/root-regionserver node of zk (ZooKeeper).

Step 2: access the-ROOT- table to find out which meta exists in the metadata information of the required data table. On the table, this. Meta. Which RegionServer is the watch on?

Step 4: visit .meta. Table to see what Region range the row key you want to query is in.

Step 5: connect the RegionServer where the specific data is located, and this step begins to query the data positively.

New version of HBase

The old version of HBase addressing has many drawbacks and has been improved in the new version. Using secondary addressing, only using the hbase:meta table to locate the region, then where to get the hbase:meta information, the answer is zookeeper. There is a / hbase/meta-region-server node in zookeeper that can get the location information of the hbase:meta table, and then query the region location information of the desired data through the hbase:meta table.

The whole query steps are as follows:

Step 1: the client first queries the location of the hbase:meta table through the / hbase/meta-region-server node of ZooKeeper.

Step 2: the client connects to the RegionServer where the hbase:meta table is located. The hbase:meta table stores the row key range information of all Region. Through this table, you can find out which Region range the rowkey you want to access belongs to, and which RegionServer the Region belongs to.

Step 3: after obtaining this information, the client can directly connect to the RegionServer that owns the rowkey you want to access and operate on it directly.

Step 4: the client will cache the meta information, and the next operation will not require the above steps to load hbase:meta.

Client API basically uses public class Example {

Private static final String TABLE_NAME = "MY_TABLE_NAME_TOO"

Private static final String CF_DEFAULT = "DEFAULT_COLUMN_FAMILY"

Public static void createOrOverwrite (Admin admin, HTableDescriptor table) throws IOException {

If (admin.tableExists (table.getTableName () {

Admin.disableTable (table.getTableName ())

Admin.deleteTable (table.getTableName ())

}

Admin.createTable (table)

}

Public static void createSchemaTables (Configuration config) throws IOException {

Try (Connection connection = ConnectionFactory.createConnection (config))

Admin admin = connection.getAdmin () {

HTableDescriptor table = new HTableDescriptor (TableName.valueOf (TABLE_NAME))

Table.addFamily (new HColumnDescriptor (CF_DEFAULT) .setCompressionType (Algorithm.NONE))

System.out.print ("Creating table.")

CreateOrOverwrite (admin, table)

System.out.println ("Done.")

}

}

Public static void modifySchema (Configuration config) throws IOException {

Try (Connection connection = ConnectionFactory.createConnection (config))

Admin admin = connection.getAdmin () {

TableName tableName = TableName.valueOf (TABLE_NAME)

If (! admin.tableExists (tableName)) {

System.out.println ("Table does not exist.")

System.exit (- 1)

}

HTableDescriptor table = admin.getTableDescriptor (tableName)

/ / Update table

HColumnDescriptor newColumn = new HColumnDescriptor ("NEWCF")

NewColumn.setCompactionCompressionType (Algorithm.GZ)

NewColumn.setMaxVersions (HConstants.ALL_VERSIONS)

Admin.addColumn (tableName, newColumn)

/ / Update column family

HColumnDescriptor existingColumn = new HColumnDescriptor (CF_DEFAULT)

ExistingColumn.setCompactionCompressionType (Algorithm.GZ)

ExistingColumn.setMaxVersions (HConstants.ALL_VERSIONS)

Table.modifyFamily (existingColumn)

Admin.modifyTable (tableName, table)

/ / disable table

Admin.disableTable (tableName)

/ / Delete column family

Admin.deleteColumn (tableName, CF_DEFAULT.getBytes ("UTF-8"))

/ / to delete a table, disable the table first.

Admin.deleteTable (tableName)

}

}

Public static void main (String... Args) throws IOException {

Configuration config = HBaseConfiguration.create ()

Config.addResource (new Path (System.getenv ("HBASE_CONF_DIR"), "hbase-site.xml"))

Config.addResource (new Path (System.getenv ("HADOOP_CONF_DIR"), "core-site.xml"))

CreateSchemaTables (config)

ModifySchema (config)

}

}

What is the role of Q1:MemStore in summarizing confusing knowledge points?

In HBase, a table can have multiple column families, a column family is physically stored together, a column family corresponds to a store, and there is a MemStore inside the store, not to improve the speed of reading and writing, but to sort the RowKey. We know that the data of HBase is stored on HDFS, but HDFS does not support modification. In order to sort by RowKey, HBase will first write the data to MemStore. The data will first be sorted into a LSM tree in Memstore, and then brushed to HFile.

In a word: the purpose of Memstore is not to speed up data writing or reading, but to maintain the data structure.

Q2: when reading data, will it be read from MemStore first?

The purpose of MemStore is to sort by RowKey, not to improve read speed. When reading data, there is a special cache called BlockCache. If BlockCache is enabled, it is to read BlockCache first, and then to read HFile+Memstore data.

What's the use of Q3:BlockCache?

Block caching (BlockCache) uses memory to record data, which is suitable for improving read performance. When block caching is enabled, HBase first queries whether there are records in the block cache, and retrieves the HFile stored on the hard disk if not.

It is worth noting that a RegionServer has only one BlockCache. BlockCache is not a necessary part of the data store, but is only used to optimize read performance.

The basic principle of BlockCache is: after reading the request to HBase, you will first try to query BlockCache, and if you can't get the data you need, go to HFile and Memstore to get it. If so, the Block block is cached in the BlockCache while the data is returned.

How does Q4:HBase delete data?

HBase deleting the record does not actually delete the data, but identifies a tombstone (tombstone marker), marking this version as invisible as well as previous versions. This is for performance reasons so that HBase can clean up these deleted records on a regular basis without having to delete them every time. The so-called regular cleaning is to do automatic merging in HBase according to a certain period of time (an operation when compaction,HBase collates and stores files, which merges multiple file blocks into a single file). In this way, the performance impact of delete operations on HBase is minimized, and even under a high concurrent load, a large number of deletions of records are OK.

There are two types of merge operations: Minor Compaction and Major Compaction.

Where Minor Compaction is the merging of multiple HFile in Store into one HFile. Data that reaches TTL during this process will be removed, but data that has been manually deleted will not be removed. The trigger frequency of this kind of merger is high.

Major Compaction merges all the HFile in the Store into a single HFile. Data that is manually deleted during this process is actually removed. At the same time, version data that exceeds MaxVersions in the cell is also deleted. The trigger frequency of this merge is low, and the default is once every 7 days. However, due to the high performance consumed by MajorCompaction, it is generally recommended to manually control the timing of MajorCompaction.

It should be noted that Major Compaction deletes data with tombstone tags, while Minor Compaction will directly ignore expired data files when merging, so expired files will be deleted when Minor Compaction.

Q5: why does HBase have high-performance reading and writing capabilities?

Because HBase uses a LSM storage structure, in the implementation of the LSM tree, the data is sorted before it is stored. LSM tree is the basic storage algorithm of Google BigTable and HBase, and it is an improvement of B+ tree of traditional relational database. The core of the algorithm is to ensure that the data is sequentially stored on disk as far as possible, and the data will be sorted out frequently to ensure its order.

A LSM tree is a bunch of small trees. The small tree in memory is called memstore. Each time flush, the memstore in memory becomes a new storefile on disk. This batch of read and write operations make the performance of HBase higher.

What is the relationship between Q6:Store and column clusters?

Region is the core module of HBase, while Store is the core module of Region. Each Store corresponds to a column family store in the table. Each Store contains a MemStore and several HFile.

Is Q7:WAL shared by RegionServer or at the Region level?

In HBase, you only need to maintain one WAL per RegionServer, and all Region objects share a WAL, instead of maintaining one WAL per Region. In this way, the log modifications caused by the update operations of multiple Region only need to be continuously appended to a single log file, and there is no need to open and write multiple log files at the same time, which can reduce the number of disk addressing and improve write performance.

However, there is a disadvantage in this approach: if the RegionServer fails, in order to recover the Region object on it, the WAL on the RegionServer needs to be split according to the Region object to which it belongs, and then distributed to other RegionServer to perform the recovery operation.

After the Q8:Master is hung up, can I still query the data?

Yes. Master service is mainly responsible for the management of table and Region. The main functions are:

Manage users' operations of adding, deleting and modifying tables to realize the splitting and merging of load balancing Region before different RegionServer to migrate the Region of failed RegionServer

When the client accesses the HBase, it does not need the participation of Master, it only needs to connect to the zookeeper to obtain the hbase:meta address, and then directly connects to the RegionServer for data reading and writing operations. Master only maintains the metadata information of the table and Region, and the load is very small. But the Master node can not be down for a long time.

These are the basic principles of HBase. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.