Shulou(Shulou.com)06/01 Report--

Ant Financial Services Group's self-developed database OceanBase went to the top of TPC-C, which caused the industry wide × × Chu to show the technical details. We specially invited OceanBase core R & D personnel to make a technical interpretation of this test, including a total of six articles:

1) introduction to TPC-C benchmark

2) how to do TPC-C testing for OceanBase

3) SQL optimization of TPC-C benchmark test

4) the challenge of database transaction engine for TPC-C benchmark

5) Storage optimization of TPC-C benchmark test

This article is the last one. For convenience of reading, we organize a series of articles into PDF e-books, follow "Ant Financial Services Group Technology" official account, reply "TPCC" and download.

Introduction

In the TPC-C standard definition, the test system is divided into two parts: RTE (Remote Terminal Emulator) and SUT. In the actual TPC-C testing process, it is not only a test of the ability of the DB side, but also a great resource consumption and pressure on all components in the link. Based on the 60.88 million tpmC test results, we have run a total of 960 RTE clients on 64 64C128G CVMs to simulate a total of 47942400 user Terminal. Finally, we need to verify the consistency and persistence based on so many RTE statistical results. SUT is divided into three parts: WAS (Web Application Server), OceanBase Proxy (OBProxy) and OceanBaseServer (OBServer). RTE's request goes to WAS, and then WAS accesses OBProxy,OBProxy through the OceanBase client, and forwards the request to the best ObServer in the back-end OceanBase cluster to execute the request. WAS and OBProxy are the bridge between RTE and OBServer, which plays a vital role in bearing pressure. In this TPC-C benchmark, there are two main components on the OceanBase access link:

ODBC interface and driver

In TPC-C testing, the WAS request OceanBase uses the ODBC interface. ODBC (Open Database Connectivity) is a data access specification proposed by Microsoft. ODBC can be used on most DBMS, and OceanBase also provides ODBC interface access. Interested users can consult ODBC API instructions to get started quickly, and users who use ODBC can directly use this API to seamlessly migrate access to OceanBase. The ODBC interface and driver are integrated into the WAS as the client requesting OceanBase.

OBProxy Agent

OceanBase implements the OBProxy proxy server to solve the routing and disaster recovery problems on the database link. OBProxy is aware of data copy address and partition rules, does not participate in the generation and scheduling of the SQL engine, and is mainly responsible for SQL routing and forwarding. In this architecture design, OBProxy undertakes the basic routing and disaster recovery functions, while all the functions of the database are implemented by ObServer. This more simple and clear division of labor can make the performance of each component more extreme, OBProxy also achieved completely stateless, only need to add nodes to achieve the horizontal expansion of agent capacity, OceanBase as a whole can also achieve the highest performance of the database.

TPC-C benchmark OceanBase Link access Graph

TPC-C is a very stringent benchmark model, which tests the full-link ability of a complete relational database system. Any link with bottlenecks can not give full play to the maximum performance of the database. Next, this paper will explain how to optimize the components on the OceanBase link in terms of performance, cost and service persistence.

Link performance optimization

Ant Financial Services Group OceanBase challenges TPCC | SQL optimization of TPC-C benchmark has been mentioned. From the point of view of the whole link, the execution time of SQL is very short, and a lot of time is spent in the process of interaction with the client, resulting in waste of resources and time-consuming increase. For this reason, OBServer provides Prepared Statement, stored procedure and ARRAY BINDING capabilities. The client and OBProxy support this capability to make it work. At the same time, the client itself also makes some optimizations to improve the link performance, and then mainly introduces the optimization points of the link performance part:

Provide asynchronous interface capability

Usually, database access is a synchronous interface, and the development of synchronous interface is convenient, but the client is greatly affected by the time-consuming network interaction, and the concurrency ability is limited. The use of multi-threading can help improve the concurrency ability, but the machine thread resources are precious, too many threads will increase the overhead of machine thread switching and limit the concurrency ability. In order to enable WAS to achieve higher throughput, we add support for asynchronous interfaces in the ODBC interface based on event-driven mechanism. Using the asynchronous interface, a single thread of WAS can continue to send requests on other Session without waiting for the execution result after sending the request, thus greatly improving the throughput capacity by making full use of thread resources. The asynchronous interface itself refers to the ODBC interface specification. If the user calls the asynchronous interface, it will be returned immediately. If the execution has not been completed, the user will return SQL_STILL_EXECUTING. The user can poll the interface until the execution is completed and return success (SQL_SUCCESS) or failure (SQL_ERROR). Based on the network event driver, you can call the API again to get the result when a result is returned. Using asynchronous interface, we can easily support a large number of concurrent connections under a small amount of thread resources, greatly improve the concurrency ability of WAS, improve the utilization of machine resources, and greatly reduce the cost of stress testing.

Provide Prepared Statement capability

PreparedStatement is a binary request interaction protocol. A PSSQL text is transferred and executed many times. The OBProxy SQL engine caches PSSQL text and parsing results. Each PSSQL only needs to perform Prepare operation once, and each subsequent execution on all Session only needs to pass in the corresponding Statement Id. The corresponding SQL parsing result can be found in the cache. Combined with the passed parameters, the routing information of OBServer can be quickly calculated. Forwarding performance is more efficient. At the same time, OBProxy, as a proxy layer, also well supports the distributed characteristics of OBServer. When Client Session needs to switch Server Session, there is no need to send the type data of PS SQL and Execute phase again. OBProxy can judge and decide whether it needs to synchronize PS SQL or add type data. Through the Prepared Statement capability, the interaction cost between systems can be effectively reduced and the performance can be improved. Compared with the ordinary SQL text interaction, a large number of SQL text transmission and CPU overhead of request text parsing are saved.

Stored procedure

For stored procedures, OBProxy does a lot of optimization, stored procedures usually contain multiple SQL, different SQL usually need to be routed to different OBServer for execution, resulting in a large number of remote execution. Remote execution will not only increase RT, but also consume more CPU resources. Therefore, OBProxy's SQL engine will parse the SQL in stored procedures, calculate the optimal policy, and send stored procedure calls to the most appropriate OBServer for execution, reducing the number of remote execution as much as possible. OBProxy also caches the parsing results and routing information of the stored procedure to save the CPU overhead caused by each hard parsing.

Complex type

We re-extend the original OceanBase transport protocol to make the whole link support complex types of transmission. At the same time, OBProxy adds parsing of complex data types, which can calculate routing and partition clipping based on complex type data. By supporting complex types, we can improve the data information carried by each transmission, effectively reduce the number of interactions, and also calculate the best routing strategy according to complex types of data, route to more OBServer in the partition as much as possible, and reduce the number of remote execution. It is with OBProxy's support for complex types such as arrays that clients can make better use of stored procedures and ARRAY BINDING capabilities.

Agent resource occupation

The OBProxy agent adopts multi-thread asynchronous framework and transparent streaming forwarding design, which ensures high-performance data forwarding (50,000 QPS per core, forwarding RT 30~50us) and minimizes the consumption of machine resources. In the TPC-C benchmark test, we also deployed to the WAS side locally, which can minimize the network overhead from the client to the agent layer. In this test, the OBProxy proxy layer is deployed to 64 client machines, and 10 Core are used on each machine, occupying a total of 640 Cores, accounting for only about 7% of the total CPU testing cost, with almost no storage resources, a small proportion of resources and supporting horizontal expansion.

Continuous service ability

OceanBase provides highly available database services. In cases where the machine is unavailable, the database can continue to provide services without any human intervention. In this Durability test of TPC-C, OceanBase shows the ability of continuous database service without human intervention, which greatly exceeds the auditor's expectations.

For OceanBase, a single machine failure scenario such as forced power outage, OBProxy has two mechanisms including blacklist and gray list, which are used to deal with OBServer peak mismatch and merge, upgrade, downtime, start / stop, network partition and other states. The blacklist is regularly refreshed and maintained, and OBServer reports back which server nodes cannot provide services. The gray list takes the initiative to trigger maintenance. When OBProxy forwards the request to OBServer, if OBServer returns a specific system error, or if OBServer is continuously unavailable several times in a period of time, the OBServer will be added to the gray list. The OBServer in the blacklist will not be accessed, and the OBServer in the gray list will retry every once in a while to check if it needs to be whitewashed to avoid a long downgrade of OBServer. Through the mechanism such as OceanBase, the continuous service ability of the database in the process of TPC-C Durability testing can be guaranteed.

Summary

OceanBase link layer provides customers with a complete end-to-end solution, and its self-developed transport protocol can flexibly support SQL features and interaction protocols, achieve a standard database access interface and support Oracle compatibility mode, and achieve the maximum balance of database ease of use, high performance and continuous service. In the follow-up, we will continue to optimize the transport protocol to achieve the highest transmission and interaction efficiency, improve the database access standard interface, and provide users with a more mature database service.

Welcome people who are interested in database drivers and protocols, high-performance servers and database middleware to join our team to build a world-class OceanBase distributed database. Please send your resume to hongwei.yhw@antfin.com

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.