Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to design the RowKey in HBase storage IM messages. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have some understanding of the relevant knowledge after reading this article.

RowKey is one of the most important aspects of HBase table design. It determines how applications interact with HBase tables and affects your performance of extracting data from HBase.

Messages in general IM systems are stored according to the TimeLine model with the session as the dimension.

The following is the RowKey design of the message store in IM system

I. format design of RowKey

Session ID and message ID are generated by snowflake algorithm, and RowKey includes three parts.

Session hash value | session id | reverse message id

Session hash values are intended for data partitioning (region) storage, and pre-partitioning can share the data read and write pressure

Session id determines a unique session, and all messages in a group have the same session id

Reverse message id determines the unique message and pulls the latest message order, and the reverse order ensures that the newer the message, the smaller the id value. In IM software, the latest message in the group is always displayed first, and then the previous message is loaded by sliding up the interface.

The purpose of "|" is to separate data bits. Because part of the id may not be consistent in length (snowflake), "|" is selected as the separation to ensure that part of the data can be solved without ambiguity.

II. Partition design of region

The characteristics of IM business determine that messages in the same session are generally read centrally (this is the case when users view a chat message). Therefore, messages for the same session need to be stored in a partition. We use the hash value of the session id as the partition field to ensure that messages for the same session must be in the same partition.

Session ID uses a modified SnowFlake algorithm (see "ID generation strategy-SnowFlake"). Session id divided by 2 to the n power, data can be distributed more evenly (if id does not have this feature, you can also choose other hash methods).

Before using HBase to store messages, messages were stored using Mysql. Using session Id%4, it is divided into 4 libraries. The figure below is as follows

HBase achieves the same partitioning effect according to rowkey, which is theoretically the same as sub-library. After the session Id is modeled, the 128values are evenly hashed to different region.

The partition value is 000-127, and the effect of pre-partition is as follows

The rowkey front 3-bit pre-partition is used. When each rowkey arrives, the first 3 bits with the same number of bits as the region are selected to match. In the example figure, 002 | … .. |... .rowkey selects 3 bits 002 for region matching, and then puts the data into the corresponding partition.

III. Message access process

1. According to the above design format, we use the session Id,002 after passing parameters | * * | * take the model 128-to distribute to different region

2. After determining the specific region, determine the unique session according to the subsequent rowkey * * | 312312312312312312312312312312 Id.

3. After determining the unique session, determine a specific message according to the follow-up message Id of rowkey. * * | 8896232141957373907. Note that this message Id has been processed in reverse order (Long.MAX_VALUE- message Id) to pull the nearest message.

On the HBase storage IM messages in how to design RowKey to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.