Shulou(Shulou.com)05/31 Report--

What is the practice of WeChat Pay trillion log in Hermes? aiming at this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

I. scale of business

At present, the maximum daily storage volume of WeChat Pay logs has reached the trillion level, and the daily storage volume has reached the PB level, while it is expected that the entire daily storage scale will further increase during the Spring Festival and other major holidays.

The Hermes cluster used by WeChat Pay's log business has also exceeded the trillion-level daily storage scale, with more than 200 nodes deployed and the total storage volume of a single cluster reaching the PB level. In addition, the daily search query concurrency is about 6000:

Under such a massive log storage scale, the entire WeChat Pay log query SLA has reached 4 9memes and 95% of the time is less than 5s.

II. Separation of deposits and accounts

The underlying storage of Hermes is implemented by HDFS, and all storage-related policies are provided by professional HDFS capabilities, including:

1. Disaster recovery of multiple copies of data

Log default storage 2 copies disaster recovery, for historical data can flexibly reduce copies to reduce storage costs, and for very important log data can also flexibly increase copies to improve data disaster recovery capability.

two。 Disk fault tolerance

Single disk or stand-alone failure HDFS can automatically migrate copies, and the whole fault tolerance process is transparent to the upper computing layer.

3. Hot and cold classification

Using the heterogeneous storage capacity provided by HDFS, combined with the daily partition storage of Hermes itself, it is very convenient to realize the hot and cold classification of data. The hot and cold graded data is transparent to the upper layer business, and the business does not need to pay attention to the storage of the data itself.

4. EC coding

EC encoding is supported after HDFS version 3.0 to further reduce storage costs, which has not been put into practice online for the time being.

By adopting this architecture of separation of deposit and calculation, on the one hand, the design of the upper computing layer can be simplified; on the other hand, when calculating the index, the computing layer only needs to calculate a single copy to achieve multi-copy disaster recovery, thus greatly reducing the CPU and memory resource consumption of the computing layer, and doubling the writing QPS.

III. Asynchronous index merging

Hermes itself uses a LSM-like data writing mode. The data is first written to memory + WAL, and then accumulated to a certain amount and then brushed to HDFS for persistent storage. In the event of a node failure, the system recovers data by rolling back the WAL, thus ensuring that the entire write is an efficient sequential write.

One of the problems brought by this efficient writing method is that with the continuous writing of data, a large number of small indexes will be generated, which will cause great pressure on the query and HDFS storage.

Hermes itself will continue to merge small indexes to reduce the number of index files; and in the night trough, we will also merge historical partition data with a larger granularity, so as to improve the query efficiency of the whole system as much as possible. The merger time of WeChat Pay's business was chosen at 2: 6 a.m., avoiding the peak of New Year's Eve red packets at 0: 1 a.m.

IV. Separation of index and data

One of the important features of business scenarios such as logs is that they are retrieved according to participle + field information, and then a complete line of logs is pulled for analysis.

In view of this scenario, the traditional column storage often has the low efficiency of obtaining row storage information, while the index and data mixed storage will have the serious problem of reading and writing IO magnification when merging indexes.

To do this, Hermes can configure to store a complete log row storage information in addition to building an index on the log by participle:

As shown in the figure above, by storing the index and data separately, only the inverted index is stored in the index catalog, and the corresponding row data of each index catalog in the same shard of row data. The result data is read in RowData through the Offset and RowId of each index directory.

Through the separation of indexes and data, the number and number of flushing of index directories are reduced by 68%, memory usage is reduced by 70%, disk usage is reduced by 14%, and retrieval performance is improved by 80%.

Fifth, hot and cold storage classification

WeChat Pay's 90% log modules are long-tailed modules with a very small amount of data. Therefore, it is very appropriate to introduce some high-performance SSD devices to speed up the query of these businesses with small storage, and in order to reduce the cost of SSD as much as possible, it is necessary to classify the business data.

The hot and cold data classification of Hermes itself is realized by using the heterogeneous storage capacity of HDFS. By configuring different replica placement strategies, the storage type used by replicas can be specified flexibly, and the whole process is transparent to the upper layer business.

The HDFS heterogeneous storage strategy is as follows:

Practice of HDFS heterogeneous storage in Hermes:

VI. Demotion of copies of historical zones

Hermes underlying storage uses HDFS multiple copies for data disaster recovery. Generally, two copies are stored by default. At present, the longest preservation period of WeChat Pay's log is 30 days, and the amount of data stored is very large.

In order to reduce the storage cost of the business as much as possible, after negotiation and communication with the business, we learned that the query demand for the log before three days is very low, and the stability of the log can be reduced a little. Therefore, the Hermes operation and maintenance side directly carries out routine copy reduction operations on the data from three days ago, thus directly reducing the entire storage cost by more than 70%. The whole copy operation is transparent to the upper computing layer and the business layer. The business doesn't know anything about it.

7. Log batch export

WeChat Pay's colleagues often have the need to batch export logs that hit certain keywords for a specified period of time:

For this reason, Hermes provides the feature of asynchronously exporting logs to HDFS and other storage media. After the user submits an export request, the system will export a copy of all the hit logs to TDW HDFS, and then drag them away with the TDW HDFS client or through the Hermes interface machine.

Log export file for users above TDW HDFS:

The answer to the question about the practice of WeChat Pay trillion log in Hermes is shared here. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.