Shulou(Shulou.com)06/02 Report--

This article introduces the relevant knowledge of "how HBase works". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

A brief overview of HBase work in Hadoop

1. Introduction

HBase is a high-reliability, high-performance, column-oriented scalable distributed storage system. It uses HBase technology to build a large-scale structured storage cluster on cheap PC servers. The goal of HBase is to store and process large amounts of data, especially large amounts of data containing thousands of rows and columns using only standard hardware configurations.

Different from MapReduce's offline batch computing framework, HBase is a random access storage and retrieval data platform, which makes up for the disadvantage that HDFS can not access data randomly.

It is suitable for business scenarios where real-time requirements are not high-HBase stores Byte arrays, which do not care about data types, thus allowing dynamic, flexible data models.

> Hadoop Ecosystem (Credit: Edureka.com)

The figure above depicts the tiers of the Hadoop 2.0 ecosystem-Hbase on top of the structured storage tier.

HDFS provides low-level storage support for HBase with high reliability.

MapReduce provides high-performance batch processing capabilities for HBase. ZooKeeper provides stable services and failover mechanisms for HBase. Pig and Hive provide high-level language support for statistical data processing. HBase,Sqoop provides available RDBMS data import functions for HDB, which makes it very convenient to migrate business data from traditional databases to HBase.

2. HBase architecture

2.1 Design Idea

HBase is a distributed database that uses ZooKeeper to manage the cluster and HDFS as the underlying storage.

At the architectural level, it consists of HMaster (the leader selected by Zookeeper) and multiple HRegionServers.

The following figure shows the infrastructure:

In the concept of HBase, a HRegionServer corresponds to a node in a cluster, a HRegionServer is responsible for managing multiple HRegion, and a HRegion represents part of the table data.

In HBase, a table may require a lot of HRegion to store data, and the data in each HRegion will not be disorganized.

When HBase manages HRegion, it defines a range of Rowkey for each HRegion. Data that falls within the defined scope is handed over to a specific area, thus distributing the load to multiple nodes, thus taking advantage of distribution and characteristics.

Similarly, HBase automatically adjusts the position of the area. If the HRegionServer is overheated, that is, a large number of requests fall on the HRegion managed by HRegionServer, HBase moves the HRegion to other nodes that are relatively idle to ensure that the cluster environment is fully utilized.

2.2 basic Architecture

HBase consists of HMaster and HRegionServer and follows the master-slave server architecture. HBase divides the logical table into multiple data blocks HRegion and stores them in HRegionServer.

HMaster is responsible for managing all HRegionServer. It does not store any data itself, but only the mapping of data to HRegionServer (metadata).

All nodes in the cluster are coordinated by Zookeeper and deal with various problems that may be encountered during HBase operation. The basic architecture of HBase is as follows:

Client: use HBase's RPC mechanism to communicate with HMaster and HRegionServer, submit requests and get results. For administrative operations, the client uses HMaster to perform RPC. For data read and write operations, the client uses HRegionServer to perform RPC.

Zookeeper: by registering the status information of each node in the cluster with ZooKeeper,HMaster, you can sense the health status of each HRegionServer at any time and avoid a single point of failure of HMaster.

HMaster: manage all HRegionServer, tell them which HRegion to maintain, and monitor the health of all HRegionServer. When the new HRegionServer logs in to HMaster, HMaster tells it to wait for the data to be allocated. When HRegion dies, HMaster marks all the HRegion it is responsible for as unassigned and then assigns them to other HRegionServer. There is no single point problem with HMaster. HBase can start multiple HMaster. Through the election mechanism of Zookeeper, there is always a HMaster running in the cluster, thus improving the availability of the cluster.

HRegion: when the size of the table exceeds the preset value, HBase automatically divides the table into different regions, each containing a subset of all rows in the table. For users, each table is a collection of data, distinguished by a primary key (RowKey). Physically, a table is divided into multiple blocks, each of which is a HRegion. We use the table name + start / end primary key to distinguish each HRegion. A HRegion stores a piece of contiguous data in a table. The complete table data is stored in multiple HRegions.

All data in HRegionServer:HBase is usually stored in HDFS from the underlying. Users can get this data through a series of HRegionServer. Typically, only one HRegionServer is running on one node of the cluster, and the HRegion for each segment is maintained by only one HRegionServer. HRegionServer is mainly responsible for reading and writing data to the HDFS file system in response to user I / O requests. It is the core module in HBase. HRegionServer manages a series of HRegion objects internally, with each HRegion corresponding to contiguous segments in a logical table. HRegion consists of multiple HStore. Each HStore corresponds to the storage of a column family in a logical table. As you can see, each column family is a centralized storage unit. Therefore, in order to improve operational efficiency, it is best to put columns with common I / O characteristics in a column series.

HStore: it is the core of HBase storage and consists of MemStore and StoreFiles. MemStore is a memory buffer. The data written by the user will be put into the MemStore first. When the MemStore is full, the Flush will be a StoreFile (the underlying implementation is HFile). When the number of StoreFile files increases to a certain threshold, the Compact merge operation is triggered, multiple StoreFile are merged into a single StoreFile, and version merge and data deletion operations are performed during the merge process. Therefore, you can see that HBase only adds data, and all updates and deletions are performed in subsequent Compact processes, so users' writes can be returned as soon as they enter memory, ensuring that HBaseI / oh when StoreFiles Compact, it will gradually form a larger and larger StoreFile. When the size of a single StoreFile exceeds a certain threshold, the split operation is triggered. At the same time, the current HRegion will be split into 2 HRegion, and the parent HRegion will be offline. HMaster assigns these two sub-HRegion to the corresponding HRegionServer to offload the load from the original HRegion to the two HRegion.

HLog: each HRegionServer has a HLog object, which is a pre-written log class that implements prewritten logging. Each time a user writes data to MemStore, it also writes a copy of the data to the HLog file. Periodically scroll and delete HLog files and delete old files (data that has been saved to StoreFile). When HMaster detects that HRegionServer is unexpectedly terminated by Zookeeper, HMaster first processes the old HLog file, splits the HLog data of different HRegion, places it in the appropriate HRegion directory, and then redistributes the invalid HRegion. During the process of loading the HRegion, the HRegionServer of these HRegion will find that the history of the HLog needs to be processed, so the data from the Replay HLog is transferred to the MemStore and then refreshed to the StoreFiles to complete the data recovery.

2.3 Root and Yuan

All HRegion metadata for HBase is stored in .meta. Watch. As the HRegion increases, the data in the .meta table increases and splits into multiple new HRegion.

In order to find the location of each HRegion in the .meta table, the metadata of all HRegion in the .meta table is stored in-ROOT-table, and finally, the Zookeeper records the location information of the ROOT table.

Before all clients can access user data, they need to access Zookeeper to get the location of-ROOT-, then access-ROOT-table to get the location of .meta table, and finally determine the location of user data based on the following information: information in the META table, as shown in the figure.

-ROOT tables are never split. It has only one HRegion, which ensures that any HRegion can be located with only three jumps. To speed up access, all areas of the. META table remain in memory

The client caches the location information of the query, and the cache does not fail actively. If the client still cannot access the data based on the cached information, the Region server of the relevant .meta table attempts to obtain the location of the data. If it still fails, ask where the .meta table associated with-ROOT-table is.

Finally, if all the previous information is invalid, Zookeeper relocates the HRegion's data. Therefore, if the cache on the client is completely invalid, you need to go back and forth six times to get the correct HRegion.

3. HBase data model

HBase is a distributed database similar to BigTable. It is a sparse long-term storage (on HDFS), multi-dimensional and sorted mapping table. The index of the table is row keyword, column keyword, and timestamp. HBase data is a string and has no type.

Think of the table as a large mapping. You can press row keys, row keys + timestamps, or row keys + columns (column families: column modifiers) to find specific data. Because HBase stores data sparsely, some columns can be empty. The above table shows the logical storage logic view of the com.cnn.www site. There is only one row of data in the table.

The unique identifier of the row is "com.cnn.www", and there is a certain amount of time for each logical modification of this data row. The tag corresponds to.

There are four columns in the table: content: HTML,anchor:cnnsi.com,anchor:my.look.ca,mime:type, and each column gives the column family to which it belongs.

The row key (RowKey) is the unique identifier of the data row in the table and is used as the primary key to retrieve the record.

In HBase, there are only three ways to access rows in a table: access through row keys, range access for given row keys, and full table scans.

Line keys can be any string (the maximum length is 64KB) and stored in dictionary order. For rows that are often read together, you need to carefully design the basic values so that they can be stored together.

4. HBase read and write process

The following figure is the HRegionServer data store diagram. As mentioned above, HBase uses MemStore and StoreFile to store updates in a table. The data is first written to HLog and MemStore after the update. The data in MemStore is sorted.

When MemStore accumulates to a certain threshold, a new MemStore is created, the old MemStore is added to the Flush queue, and a separate thread is flushed to disk to become StoreFile. At the same time, the system records a CheckPoint in Zookeeper, indicating that data changes before that time have been retained. Data in MemStore may be lost when an accident occurs on the system.

In this case, HLog is used to recover data after CheckPoint.

StoreFile is read-only and cannot be modified once created. Therefore, the update of HBase is an additional operation. When the StoreFile in the store reaches a certain threshold, the merge operation is performed and the modifications of the same key are merged to form a large StoreFile. When the size of the StoreFile reaches a certain threshold, the StoreFile is split into two StoreFiles.

4.1 write operation flow

Step 1: the client sends a write data request to HRegionServer through the scheduling of Zookeeper, and writes the data to HRegion.

Step 2: write the data to HRegion's MemStore until MemStore reaches the preset threshold.

The data in step 3:MemStore is sorted out into StoreFile.

Step 4: as the number of StoreFile files increases, when the number of StoreFile files increases to a certain threshold, the Compact merge operation is performed, multiple StoreFiles are merged into one StoreFile, and version merging and data deletion are performed in the version library. meanwhile.

Step 5:StoreFiles gradually forms a larger and larger StoreFile through continuous Compact operations.

Step 6: after the size of a single StoreFile exceeds a certain threshold, the Split operation is triggered to split the current HRegion into two new HRegion. The parent HRegion will be offline and the two child HRegion of the new Split will be assigned by the HMaster to the corresponding HRegionServer so that the pressure from the original HRegion can be offloaded to the two HRegion.

4.2 read operation flow

Step 1: the client accesses Zookeeper, finds-ROOT-table, and gets .meta. Table information.

Step 2: search from .meta. The table gets the HRegion information of the target data and finds the corresponding HRegionServer.

Step 3: get the data you need to find through HRegionServer.

The memory of step 4:HRegionserver is divided into two parts: MemStore and BlockCache. MemStore is mainly used to write data, while BlockCache is mainly used to read data. First read the request to MemStore to check the data, check the BlockCache check, then check the StoreFile, and then put the read results into the BlockCache.

5. HBase usage scenarios

Semi-structured or unstructured data: for poorly defined or confusing data structure fields, it is difficult to extract data according to the concepts applicable to HBase. If more fields are stored as the business grows, RDBMS needs to be closed to maintain the change table structure, and HBase supports dynamic addition.

Records are very sparse: how many columns of RDBMS rows are fixed, while empty columns waste storage space. Columns with empty HBase are not stored, which saves space and improves read performance.

Multi-version data: values located based on RowKey and column identifiers can have any number of version values (timestamps are different), so it is convenient to use HBase for data that needs to store change history.

Large amount of data: as the amount of data becomes larger and larger, the RDBMS database will not be able to bear it, and there is a read-write separation strategy. Through one host, it is responsible for the write operation, while multiple slaves are responsible for the read operation, which doubles the server cost. As the pressure increased, the captain could not bear the pressure. At this point, the library is divided and the almost unrelated data is deployed separately. Some join queries cannot be used and require the use of the middle tier. With the further increase of the amount of data, the record of the table becomes larger and larger, and the query becomes very slow.

Therefore, it is necessary, for example, to model the ID to divide the table into multiple tables to reduce the number of records in a single table. People who have experienced these things know how to abandon the process.

HBase is simple, as long as a new node is added to the cluster, HBase is automatically split horizontally, and seamless integration with Hadoop ensures data reliability (HDFS) and high-performance massive data analysis (MapReduce).

6. HBase Map Reduce

The relationship between Table and Region in HBase is similar to that between File and Block in HDFS. Because HBase provides API to interact with MapReduce, such as TableInputFormat and TableOutputFormat, HBase data tables can be directly used as the input and output of Hadoop MapReduce, which is beneficial to the development of MapReduce applications and does not need to pay attention to the processing of HBase. Details of the system itself.

This is the end of the introduction to how HBase works. Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.