Shulou(Shulou.com)06/03 Report--

Difference

There are many ways to do nosql, hive and hbase, both of which are based on hadoop. The difference is that hive is based on a sql-like engine, so you can use HSQL statements to count queries. It is slow to return data, and all the data in the convenience table is the default.

Hbase is the database of key/value. It is good at real-time query. It is used for real-time message analysis of facebook.

Noun: MapReduce

Master assigns tasks to worker to execute. The map function reads the assigned input data fragments and outputs a collection of intermediate key/value pair values. The reduce function phone has a value value of the middle key value in the phase, and combines these values to form a smaller set of values.

Classify some data through map, and process the same kind of data through reducer.

Hbase should understand the following modules

Simply understand that hbase updates data, first write to hlog log, then hregionserver to region at the same time, then store through memstore of store module, and execute store file,flush to hfile if the set value is reached, so that all these modules are used

Master or hmaster is the implementation of Master Server, which is responsible for monitoring RegionServer instances in the cluster and is also the interface for all metadata changes. In the cluster, it usually runs on NameNode.

Hregionserver is the implementation, service and management Regions of RegionServer. RegionServer in the cluster runs on DataNode.

Region represents table,Region with multiple Store (column clusters), Store has one Memstore and multiple StoreFiles (HFiles), and the underlying layer of StoreFiles is Block

Connect https://blog.csdn.net/maketubu7/article/details/80612930

Https://blog.csdn.net/huangliuyu00/article/details/79704371

Hlog

Memstore

Store file

How does Hlog ensure hbase consistency

The reliability of HBase is realized by HLog, and the HLog mechanism is an implementation of WAL (Write-Ahead Logging), which writes logs in advance. WAL is a common way to implement consistency in transaction mechanisms. There is a HLog implementation in each Region Server. During the Region Server update operation, the record is first written to the WAL (that is, HLog), and then to the MemStore of the Store. Finally, the MemStore will write the data to the persistent HFile, when the MemStore reaches the configured memory threshold. The reliability of writing is guaranteed by first writing WAL (HLog) and then writing MemStore.

If there is no WAL, when Region Server is down, MemStore is not written to HFile, or StoreFile is not saved, then the data will be lost. While the data of HFile is guaranteed by HDFS, there will be 3 copies in the default HDFS.

If the server crashes, region server will call back to Hlog to recover the server's in-memory data through data writeback. The following figure shows the writing process, which is also a frame diagram of Hbase

-

In addition, what is the definition of cell,column family, columns, table,row

The connection is clear, https://blog.csdn.net/huangliuyu00/article/details/79704371.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.