Shulou(Shulou.com)06/02 Report--

Track 3:Application

Track3 is about the sharing of HBase applications. Engineers from Tencent, Kuaishou, Didi, Pinterest, China Mobile, China Life and other companies shared the problems and experiences encountered in HBase application and practice for us.

Track3-1: HBase at TencentPPT download link: http://t.cn/AijgoTGY

Cheng Guangxu, an engineer from Tencent, brings us the application scenarios and experience of HBase in Tencent's business.

Tencent currently has more than 90 HBase clusters, the largest of which has more than 500 nodes. Many businesses within Tencent, including Tencent Video, WeChat Pay and Tencent Cloud, are using HBase services. First of all, it shares the experience of data migration using HBase: Replication and ExportSnapshot. In practical use, the amount of data in the business is very large every day, and the period in which these data need to be saved is either very large or very small. Therefore, it adopts the method of day-by-day table, that is, a new table is created every day, and for out-of-date data, the table for the day can be deleted directly. Secondly, the optimization of bandwidth is shared.

There are five main parts of the traffic written to HBase:

1. Write, 2.WALJournal 3.Flush Compaction,5.Major Compaction 4.Small. The optimization methods are: 1. Enable compression of CellBlock. Compression of 2.WAL. 3. Increase memstore, decrease Flush, decrease Compaction. 4. Reduce the number of threads in Compaction. 5. Close Major Compaction. 6. Build the table by day. Finally, it introduces how to share RestServer. When setting up a RestServer for each HBase cluster, if there are few requests to read the cluster, then the RestServer resources of the cluster are wasted. Tencent has made an improvement by configuring one RestServer to access multiple HBase clusters and noting in MySQL which tables can be accessed in this way.

Track3-2: HBase at Kuaishou

PPT download link: http://t.cn/AijgodXA

Xu Ming, an engineer from Kuaishou, shared the application and practice of HBase in Kuaishou.

Kuaishou has a large number of users uploading a large number of videos every day, most of which are MB objects. The storage scheme is that the data is directly stored on HDFS, the index of data is stored on HBase, and the latest data is stored in memcache.

In order to improve the availability of HBase and speed up fault recovery, Kuaishou developed a hawk system, which includes three components of master,agent,sniffer, in which multiple agent votes to confirm whether a node is dead, and then sniffer handles the hanging node, and deletes the problematic node from the zk quickly. At the same time, Kuaishou made some optimizations to speed up the process of split log and assign region. On the client side, the main task is to ensure that the region location of the problematic node can be quickly cleaned up.

The RSGroup function is also heavily used in Kuaishou, and some optimizations have been made. One is to add FallBack RSGroup, if all the nodes of a RSGroup are down, select the machine from this FallBack RSGroup; the other is to add Global RSGroup, mainly to meet the monitoring needs, because the canary table of hbase needs to be distributed on different machines.

Kuaishou also shared examples of how it uses HBase to store and analyze huge amounts of data. For example, to solve the problem of computing user retention, if you use SQL, the execution is very slow, and Kuaishou uses Bitmap's solution. Convert the dimensions that need to be extracted into Bitmap and use space to reduce the time consumed. Use MR to convert the selected dimension into Bitmap, cut the Bitmap into small pieces, and import the Bitmap data and meta into HBase.

Track3-3: HBase at DiDi

PPT download link:

Http://t.cn/AijgK2qq

Tang Tianhang, an engineer from Didi, brings us the application scenarios and experience of HBase in Didi's business.

Didi has 7 domestic HBase clusters and 4 overseas international clusters. It covers all the business lines of Didi. At present, it serves more than 200 items, and the data level is PB.

There are two main scenarios for Didi to use HBase: 1. Offline data query, including the use of Phoenix, Kylin, openTSDB, and the trajectory data system constructed by 2.GeoMesa system, can be used for real-time query, monitoring and feature engineering data mining. GeoMesa system provides import and export interfaces, import interfaces support Native API, MR/BulkLoad, StreamingSQL, export interfaces support SparkCore, SparkSQL, CQL, GeoServer. This use of GeoMesa can have the following benefits: 1, out of the box, 2, SQL-like text language support, 3, horizontally scalable, 4 based on Hadoop ecology.

Didi's improvement to zookeeper in practice is to separate the ZK that server and client depend on, so that when the zk is unavailable due to the sudden and large number of visits on the client, it will not affect the server. (HBASE-20159,ZOOKEEPER-832). Didi's improvements on HBase/Phoenix are mainly Quota settings, replication and query optimization (HBASE-21964,HBASE-22620,PHOENIX-5242).

Finally, Didi established a multi-user full-link tracking from Client to HAProxy, then to Thriftserver and QueryServer, and then to Hbase, which can more effectively improve the efficiency of operation and maintenance.

Track3-4: Phoenix Best Practice In China Life Insurance Company

PPT download link: http://t.cn/AijgKfM4

Yuan Liou, an engineer from China Life Insurance, shared the best practices of Phoenix in China Life Insurance.

At present, the total number of nodes of China Life is more than 200, and that of Phoenix cluster is more than 30. The overall data volume of the cluster is 1300T HBASE, with a maximum of 30T in a single table, and there are about hundreds of scripts running every day.

Phoenix application scenario in China Life: the data source is generated from the core trading system, and then through SharePlex, call Kafka, data from Kafka real-time access to the Phoenix cluster, through the query service to provide access to rights and interests information for APP. In terms of physical architecture, the bottom layer is the Phoenix cluster, with two links up, one is Phoenix Gateway and the other is real-time query service, which is connected to the Weblogic cluster through load balancing.

Yuan Liou introduced the design of Spark Streaming: 1, for the integrated table, some control fields will be added to record the update time, delete or insert operation and so on. 2. Real-time synchronization program, which is distinguished by table name or statistical field.

Yuan Liou then introduced the optimization of Phoenix, taking the system table of Phoenix as one group and the data table in another group. When the client accesses, the metadata is pulled once a day, and then you don't have to access the Phoenix system table, which reduces the load. Some optimizations based on HBase include: 1. Region Balance By Table. 2 、 G1GC

3 、 Manual MajorCompaction

Track3-5: HBase Practice in China Mobile

PPT download link: http://t.cn/AijgOxGa

Chen Yechao, head of Hbase from China Mobile Suzhou R & D Center, introduced the practice of Hbase in China Mobile.

China Mobile currently has about 6000 physical nodes, more than 100 clusters, dozens of PB data, a single cluster maximum of more than 600 nodes, a single table maximum 1.6PB, a maximum of 30 million concurrent access, and the stored data is compressed with a high compression ratio to reduce storage space.

Several application scenarios of HBase in China Mobile: 1. Beijing Mobile's traffic list, such as mobile phone usage, can be found in the handheld business hall. 2. DPI data, mainly related to signaling, has some network optimization design. 3. Monitoring and logging, including small pictures, user tags, crawlers and marketing, etc.

China Mobile solves the problem of data skew in BulkLoad through data sampling in practice. Data compression is not enabled in the Flush and BulkLoad phases, but is only used in the compaction phase to improve read and write performance. SSD/HDD disk is mixed, and the data is stored in HDD disk after compaction. For better use of SSD, China Mobile has done the following work: 1.2.6 version of SSD backport HSM To HBase. 2. The write path of all users is SSD, and the write performance is improved by 50%. In addition, China Mobile has also developed HBase cluster intelligent operation and maintenance tools: Slider and RegionServerGroup, which can control the allocation of resources, and has made a set of authority authentication system based on Region.

Track3-6: Recent work on HBase at Pinterest

PPT download link: http://t.cn/AijgO0KU

Xu Lianghong, technology lead from Pinterest, shared the latest development of HBase in Pinterest.

Pinterest currently has 50 clusters, all of which are deployed on AWS, and the amount of data is about PB. HBase 0.94 was used in 2013 and upgraded to version 1.2 in 2016.

Pinterest realizes the transaction support of HBase through Apache Omid, and finds that there is a performance bottleneck in Omid. Then we develop the Sparrow system by ourselves. The main improvements are as follows: 1, move the commit operation to the client to solve the Transaction Manager single point problem. 2. Change Transaction Manager to multi-threaded implementation, so that begin operations do not have to wait for commit to complete. Compared with Omid, the P99 delay of Sparrow is 100 times lower in Begin phase and 3 times lower in commit phase.

Pinterest developed the Argus system, which is combined with Kafka to provide WAL notification mechanism. The approximate implementation is as follows: the data that needs notification mechanism will be tagged when client is written, and these tags will be passed to the WAL level, and the WAL will be provided to Argus Observer for data processing through Kafka, which is user-defined.

Pinterest implements Ixia based on open source Lily, which is used to build HBase secondary index in real time. At the same time, it integrates Muse to realize SQL-like query. Approximate implementation: the data written into HBase will be transferred to Replication Proxy, and when called to Indexer through Kafka, index manager will read the columns of HBase data. If you need to build an index, the data will be written into Muse. Muse will retrieve the data according to the schema of the data, query will query it in Muse, and HBase will be queried when needed.

Xu Lianghong introduced the benefits of Argus and Ixia design: 1. The asynchronous replication mechanism has little impact on writes. 2, independent of the HBase system, run separately, can quickly carry on the data processing.

Track3-7 HBase at Tencent Cloud

PPT download link: http://t.cn/AijgOeGJ

Chen long, an engineer from Tencent, shared HBase's experience on Tencent Cloud.

Cloud services will encounter a lot of management and user-related problems. Chen long explained three challenges of cloud services: 1. A lot of technical consulting work. 2. Handling of emergency situations. 3. Fault location analysis. And combine two cases to analyze the challenges of cloud services.

In terms of monitoring, Tencent Cloud collects the metirc of table and region through OpenTSDB. Users can log in to Cloud Monitor, set Qps to a certain threshold, and then make reverse notification.

Chen long analyzed the four reasons for the failure on the cloud:

1. External factors, such as resource leakage, large number of requests, coprocessor problems

2. Hardware factors, disk, network, CVM, resources

3. Storage factors, block loss, read and write timeout

4. Configuration factors: jvm, memstore, blockcache, flushsize

Tencent Cloud solves a variety of problems encountered in the cloud by providing documentation, tools and monitoring. Chen long finally shared the architecture of the monitoring system. Share the architecture of cloud management services, such as the need for rapid capacity expansion or cluster reduction.

Please indicate the source of the reprint "Xiaomi Cloud Technology".

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.