Shulou(Shulou.com)06/03 Report--

This article goes on to talk about HBase.

4. Hbase fault tolerance and recovery

There is a HLog object in each HRegionServer. HLog is a class that implements Write Ahead Log. Each user operation writes a piece of data to the HLog file (see the following HLog file format). The HLog file periodically scrolls out the new file and deletes the old file (data that has been persisted to the StoreFile). When HRegionServer terminates unexpectedly, HMaster will perceive through Zookeeper that HMaster will first process the legacy HLog file, split the Log data of different Region into the corresponding region directory, then redistribute the invalid region, and then get the HRegionServer of these region. In the process of Load Region, you will find that there is a historical HLog to deal with, so the data in Replay HLog will be sent to MemStore, and then flush to StoreFiles to complete data recovery.

Hbase fault tolerance:

1) Master fault tolerance: Zookeeper reselects a new Master

L in the process without Master, data reading is still going on as usual

L region segmentation and load balancing cannot be carried out in the process without master

2) RegionServer fault tolerance: regularly report the heartbeat to Zookeeper. If the heartbeat does not occur within a period of time, Master reassigns the Region on the RegionServer to other RegionServer, and the "pre-write" log on the failed server is split by the master server and sent to the new RegionServer.

3) Zookeeper fault tolerance: Zookeeper is a reliable service, generally configured with 3 or 5 Zookeeper instances

Region positioning process:

Search for RegionServer process: ZooKeeper-- >-ROOT- (single Region)-- > .meta.-- > user table

1)-ROOT-

L table contains .meta. The region list where the table is located, and the table will have only one Region

The location of the-ROOT- table is recorded in the Zookeeper.

2) meta.

The table contains all the user space region lists, as well as the server address of the RegionServer.

5. Basic operation of Hbase

1) enter hbase shell console

$HBASE_HOME/bin/hbase shell

Management of tables:

2) check which tables are available

List

3) create a table

# Syntax: create, {NAME = >, VERSIONS = >}

# for example: create table T1 with two family name:f1,f2, and the number of versions is 2

> create't 1x, {NAME = > f1m, VERSIONS = > 2}, {NAME = > f2m, VERSIONS = > 2}

4) delete the table

# in two steps: first disable, then drop

# for example: delete table T1

> disable 't1'

> drop 't1'

5) View the structure of the table

# Syntax: describe

# for example: view the structure of table T1

> describe 't1'

6) modify the table structure

# disable is required to modify the table structure

# Syntax: alter 't1cards, {NAME = > 'f1'}, {NAME = >' f2', METHOD = > 'delete'}

# for example: modify the TTL of the cf of table test1 to 180 days

> disable 'test1'

> alter 'test1', {NAME= >' body',TTL= > '15552000'}, {NAME= >' meta', TTL= > '15552000'}

> enable 'test1'

Rights Management:

1) assign permissions

# Syntax: grant parameters are separated by commas

# permissions are denoted by five letters: "RWXCA".

# READ ('R'), WRITE ('W'), EXEC ('X'), CREATE ('C'), ADMIN ('A')

# for example, assign user 'test' access to read and write to table T1

> grant 'test','RW','t1'

2) View permissions

# Syntax: user_permission

# for example, view the permission list of table T1

> user_permission 't1'

3) take back the authority

# similar to assigning permissions, syntax: revoke

# for example, revoke the permissions of the test user on table T1

> revoke 'test','t1'

Addition, deletion, modification and query of table data:

1) add data

# Syntax: put

# for example: add a row to table T1: rowkey is rowkey001,family name:f1,column name:col1,value:value01,timestamp: system default

> put 't1 recording, rowkey001, fueling, col1, recording, value01'

2) query data-query a row of records

# syntax: get, [,....]

# for example: query the value of col1 under F1 in T1 ~ ~ Rowkey 001

> get 't1 recording, "rowkey001", "f1ambigucol1"

# or:

> get 't1 recording, {COLUMN= >'F1 col1'}

# query all column values under F1 in table T1

Hbase (main) > get 't1century recording rowkey001'

3) query data-scan table

# syntax: scan, {COLUMNS = > [,.... ], LIMIT = > num}

# in addition, advanced features such as STARTROW, TIMERANGE and FITLER can be added

# for example: scan the first five pieces of data in table T1

> scan't 1 percent, {LIMIT= > 5}

4) query the number of rows in the table

# Syntax: count, {INTERVAL = > intervalNum, CACHE = > cacheNum}

# INTERVAL sets the number of rows to be displayed and the corresponding rowkey. By default, the size of the cache to be fetched by 1000 cache is 10. Adjust this parameter to improve the query speed.

# for example, the number of rows in the query table T1 is displayed every 100, and the cache area is 500

> count't 1 million, {INTERVAL = > 100, CACHE = > 500}

5) Delete data-delete a column value in a row

# Syntax: delete, must specify a column name

# for example: delete the data of f1:col1 in table T1 ~ ~ Rowkey 001

> delete 't1recording, recording rowkey001recording, recording, writing, writing, etc.

6) Delete data-delete rows

# Syntax: deleteall, you can delete the entire row of data without specifying the column name

# for example: delete the data of table T1 ~ rowk001

> deleteall 't1recording and recording rowkey001'

7) Delete data-delete all data in the table

# Syntax: truncate

# the specific process is: disable table-> drop table-> create table

# for example: delete all data in table T1

> truncate 't1'

Region Management:

1) Mobile Region

# syntax: move 'encodeRegionName',' ServerName'

# encodeRegionName refers to the coding after regioName, and ServerName refers to the Region Servers list of master-status

# exampl

> move '4343995a58be8e5bbc739signals,' db-41.xxx.xxx.org,60020139'

2) enable / disable region

# Syntax: balance_switch true | false

Hbase (main) > balance_switch

3) Manual split

# syntax: split 'regionName',' splitKey'

4) manually trigger major compaction

# Syntax:

# Compact all regions in a table:

> major_compact 't1'

# Compact an entire region:

> major_compact 'r1'

# Compact a single column family within a region:

> major_compact'R1', 'c1'

# Compact a single column family within a table:

> major_compact 't1century, 'c1'

At this point you have learned to install hadoop clusters, understand the HDFS file system, MapReduce computing framework and Zookeeper collaboration services (Zookeeper data model, access control, application scenarios), today finished learning HBase, the next article we introduce hadoop's database tool-Hive.

How to learn Hadoop development in 4 months and find a job with an annual salary of 250000?

Free to share a set of 17 years of the latest Hadoop big data tutorials and 100 Hadoop big data must meet questions.

Because links are often harmonious, friends who need them please add Wechat ganshiyun666 to get the latest download link, marked "51CTO"

The tutorials have helped 300 + people successfully transform Hadoop development, with a starting salary of more than 20K, double the previous salary.

Baidu Hadoop core architect recorded it himself.

The content includes three parts: basic introduction, Hadoop ecosystem and real business project. Among them, business cases allow you to come into contact with the real production environment and train your development skills.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.