Shulou(Shulou.com)06/02 Report--

From: instant messaging network

Http://www.52im.net/thread-2141-1-1.html

Founded in May 2014, Dada, with business coverage in 37 cities across the country, has 1.3 million registered crowdsourced distributors and delivers 1 million pieces per day, making it the country's leading logistics and distribution platform for the last three kilometers. Dada's business model is very similar to Didi and Uber, using social idle human resources in a crowdsourced way to solve the problem of real-time distribution of the last three kilometers of O2O (Dada was merged with JD.com in April 2016).

The composition of Dada's business is simple and direct-merchants place orders, dispatchers take orders and distribution, and it is precisely because it is easy to understand that Dada's business can achieve explosive growth in a short period of time. Behind the rapid growth of the business is the result of the continuous rapid technology iteration of the Dada technical team. This article takes this opportunity to summarize and share the first-hand practical data of this series of technological evolution. I hope it can inspire you who are also struggling in the front line of Internet entrepreneurship.

Technical background

The Dada business mainly consists of two parts:

Merchants issue orders

The dispatcher accepts the order for distribution.

Dada's business logic looks very simple and straightforward, as shown in the following figure:

The scale of Dada's business has grown tremendously, from zero to nearly one million orders a day in about a year, putting great pressure on the back end. Pressure is mainly divided into two types: reading pressure and writing pressure. The reading pressure comes from the dispatcher scrambling for orders in APP, high-frequency refresh query around the order, hundreds of millions of visits per day, the peak QPS up to thousands of times per second. Writing pressure comes from merchants issuing orders, Dada receiving orders, picking up goods, completion and other operations. The pressure of reading in Dada business is much greater than that of writing, and the number of read requests is more than 30 times that of writing requests.

The following picture shows the daily traffic trend of Dada in the early stage of its growth, which shows that it is growing very fast:

The following figure shows the changing trend of Dada's request for QPS in the early and peak periods of growth. It can be seen that the growth is extremely fast:

With the rapid growth of the business, the demand for technology is getting higher and higher. we must make full preparations in the architecture in order to meet the challenges of the business. Next, let's take a look at how Dada's background architecture has evolved.

Original technical architecture: simple and straightforward

As a start-up, the most important thing is agility, rapid implementation of products and external services, so we chose public cloud services to ensure rapid implementation and scalability, saving time such as self-building computer rooms. In terms of technology selection, in order to respond quickly to business requirements, the business system uses Python as the development language and the database uses MySQL.

As shown in the following figure, several major systems in the application layer access a database:

Medium-term architecture optimization: the bottleneck of read-write separation database is becoming more and more serious

With the development of business and the rapid growth of traffic, the above solutions can not meet the performance requirements quickly: the response time of each request is getting longer and longer, for example, the dispatcher refreshes the surrounding order in app, and the response time increases from the initial 500ms to more than 2 seconds. During the peak of business, the system even experienced downtime, and some merchants and distributors even doubted the quality of our service. At this critical moment of life and death, through monitoring, we found that the peak MySQL CPU utilization rate was close to 80%, and the disk IO utilization rate was close to 90%. Slow Query increased from 100 to 10,000 per day, and became more and more serious day by day. The database has become a bottleneck, and we have to upgrade the architecture quickly.

The following is the qps change chart of the database for a week, which shows that the pressure on the database is increasing rapidly:

Our read-write separation scheme

When there is a performance bottleneck in the Web application service, because the service itself is stateless (stateless), we can solve it by adding machines to scale horizontally. Obviously, the database can not be expanded by simply adding machines, so we adopt the scheme of MySQL master-slave synchronization and application server read-write separation.

MySQL supports master-slave synchronization, copying data incrementally from the master library to the slave database in real time, and one master library can connect multiple slave libraries for synchronization.

Taking advantage of this feature of MySQL, we judge the read and write of each request on the application server:

If the request is written, all DB operations in the request are sent to the main database

In the case of a read request, all DB operations in this request are sent to the slave library.

After the separation of read and write, the pressure on the database is much reduced, CPU usage and IO usage are reduced to less than 5%, and Slow Query is close to 0.

Master-slave synchronization and read-write separation bring us the following two main benefits:

Reduce the pressure on the master library (write): Dada's business mainly comes from the read operation. after the separation of read and write, the read pressure is transferred to the slave database, and the pressure on the master library is reduced tens of times.

From the library (read) can be extended horizontally (plus the slave machine): because the system pressure is mainly read requests, and the slave library can be extended horizontally, when the slave pressure is too high, you can directly add the slave machine to alleviate the read request pressure.

The following is the change diagram of the optimized database QPS:

Select QPS of the main library before read-write separation

Select QPS of the main library after read-write separation

New situation: master-slave delay problem

Of course, no plan is omnipotent.

The separation of reading and writing temporarily solves the problem of MySQL stress, but also brings new challenges:

At the peak of business, the merchant sends out the order, but I don't see the order in my order list (typical read after write)

There are occasional exceptions within the system that cannot query the data.

Through monitoring, we found that master-slave delay may occur in MySQL during the business peak. In extreme cases, the master-slave delay is as high as 10 seconds.

How to monitor the master-slave synchronization status? On the slave machine, execute show slave status to check the Seconds_Behind_ Master value, which represents the time (in seconds) that the master and slave synchronization lags behind the master library. If there is no delay in synchronization with the slave, this value is 0. One of the important reasons for MySQL master-slave delay is that master-slave replication is executed in a single thread.

So how to avoid or solve the master-slave delay? We have made some optimizations as follows:

Optimize MySQL parameters, such as increasing innodb_buffer_pool_size, allowing more operations to be done in MySQL memory and reducing disk operations

Use high-performance CPU hosts

Physical hosts are used in the database to avoid virtual hosts and improve IO performance.

Use SSD disks to improve IO performance. The random IO performance of SSD is about 10 times that of SATA hard disk.

Business code optimization, the real-time requirements of some operations, the use of the main library to do read operations.

The write operation of the main library is getting slower and slower.

The separation of reading and writing can solve the problem of reading pressure very well. each time the reading pressure increases, it can be extended horizontally by adding from the library. However, with the explosive growth of business, there is no effective way to alleviate the pressure of write operation, such as the slower the merchant issues the order, which seriously affects the business experience. Our monitoring found that database writes are getting slower and slower, and a normal insert operation may even be performed for more than a second.

It can be seen that disk IO utilization is already very high, with peak IO response time up to 636ms and IO utilization up to 100%.

At the same time, the business is becoming more and more complex, multiple application systems use the same database, and one of the small non-core functions appears Slow query, which often affects other core business functions on the main database.

We have an application system that records logs in MySQL. The log volume is very large, nearly 100 million rows are recorded, and the ID of this table is UUID. One day during the peak period, the whole system suddenly slows down, which leads to downtime. Monitoring found that the table insert was so slow that it slowed down the entire MySQL Master, which in turn dragged across the entire system. Of course, keeping a journal in MySQL is not a good design, so we developed big data's log system. On the other hand, UUID as the primary key is a bad choice. In the following horizontal sub-library, there is a more in-depth description of ID generation.

Further split the main library to optimize the slow write operation of the main library

At this point, the main library became a performance bottleneck, and we realized that we had to upgrade the architecture again and split the main library:

On the one hand, to improve performance

On the other hand, reduce the interaction between systems to improve the stability of the system.

This time, we split the system vertically by business.

As shown in the following figure, the initial huge database is split into different business databases by business, and each system only accesses the corresponding business database to avoid or reduce cross-database access:

The following figure shows the pressure on the main database after the vertical split. You can see that the disk IO utilization has been greatly reduced. The peak IO response time is 2.33ms, and the highest IO utilization is only 22.8%:

The future is bright and the road is tortuous.

The process of vertical separation of libraries also meets many challenges, the biggest challenge is that it is not possible to join across libraries, and it is necessary to ReFactor the existing code. When you use a single database, you can simply use join associated table query; after splitting the database, the split database on different instances, you can not use join across libraries.

For example, in the CRM system, you need to query all orders of a merchant through the merchant name. Before dividing the database vertically, you can query the merchant and order table, as shown below:

After dividing the library, you need to ReFactor the code. First, query the merchant id through the merchant name, and then query the order table through the merchant Id, as shown below:

The lessons learned in the process of vertical library separation have led us to develop SQL best practices, one of which is to disable or reduce the use of join in the program, but should assemble data in the program to make SQL easier. On the one hand, it prepares for further vertical split of services, on the other hand, it avoids the problem of low performance of join in MySQL.

After a week of intensive underlying architecture adjustment and business code refactoring, the vertical split of the database is finally completed. After the split, each application only accesses the corresponding database. On the one hand, it splits the single point database into several, sharing the writing pressure of the main database; on the other hand, the split database is independent, realizing business isolation and no longer affecting each other.

Prepare for the future and further upgrade the architecture: horizontal sub-library (sharding)

Through the sharing in the previous section, we know:

The separation of reading and writing solves the reading pressure by expanding from the library.

Vertical split caches the write pressure by splitting the main library by business.

But does the technical team have peace of mind? The answer is NO.

The above architecture still has the following hidden dangers:

The amount of data in a single table is getting larger and larger: for example, the number of records in a single table will soon exceed 100 million, exceeding the limit of MySQL and affecting the performance of reading and writing.

The writing pressure of the core business library is increasing: it can no longer be split vertically, and the MySQL main library does not have the ability to scale horizontally.

In the past, system pressure forced us to upgrade our architecture, but this time, we need to upgrade our architecture in advance to achieve horizontal expansion of the database (sharding). Our business is similar to Uber, while Uber did not implement horizontal repositories until 5 years after the establishment of the company (2014), but our business development requires us to start implementing horizontal repositories 18 months after its establishment.

The first question facing horizontal split is the logic to split it:

One solution is to split by city, with all the data of a city in a database.

Another option is to split the data averagely by order ID

The advantage of splitting by city is that the degree of data aggregation is relatively high, the aggregation query is relatively simple, and the implementation is relatively simple, but the disadvantage is that the data distribution is uneven, and the amount of data in some cities is very large, resulting in hot spots, which may be forced to be split again in the future.

Split by order ID is on the contrary, the advantage is that the data is evenly distributed, there will not be a database data is very large or very small, the disadvantage is that the data is too scattered, it is not conducive to aggregate query. For example, after splitting by order ID, a merchant's orders may be distributed in different databases. To query all orders of a merchant, you may need to query multiple databases. In view of this situation, one solution is to make redundant tables of the data that need to aggregate queries, and the redundant tables are not split. At the same time, in the process of business development, aggregate queries are reduced.

After weighing the pros and cons repeatedly and referring to the sub-library scheme of Uber and other companies, we finally decided to divide the library horizontally according to the order ID.

Architecturally, we divide the system into three layers:

Application layer: all kinds of business application systems

Data access layer: unified data access interface, shielding technical details such as read-write sub-library, sub-library, cache and so on to the upper application layer.

Data layer: slicing DB data, and dynamically adding shard shards.

The technical key point of horizontal database division lies in the design of data access layer.

The data access layer mainly consists of three parts:

ID generator: generate the primary key for each table

Data source routing: route each DB operation to a different shard data source

Caching: Redis is used to cache data to improve performance.

The ID generator is the core of the entire horizontal library, which determines how to split the data, and how to query, store-retrieve data:

ID needs to be globally unique across libraries, otherwise it will cause conflicts in the business layer.

In addition, the ID must be numeric and ascending, mainly considering that the ascending ID can guarantee the performance of MySQL.

At the same time, the ID generator must be very stable because any failure will affect all database operations

Our ID generation strategy draws lessons from Instagram's ID generation algorithm.

As shown in the figure above, the solution is described as follows:

The binary length of the entire ID is 64 bits.

The first 36 bits use timestamps to ensure that ID is added in ascending order

The middle 13 bits are sub-library identifiers, which are used to identify the database in which the records corresponding to the current ID are in.

The last 15 bits are MySQL self-increasing sequences to ensure that ID will not repeat when concurrency occurs within the same second. Each shard library has a self-increment sequence table. When generating a self-increment sequence, the current self-increment sequence value is obtained from the self-increment sequence table and added 1 as the last 15 bits of the current ID.

Write at the end

Entrepreneurship is a race against time. In order to meet business needs quickly, we adopt simple and efficient solutions, such as using cloud services and application services to directly access a single point of DB.

In the later stage, with the increase of system pressure, performance and stability are gradually taken into consideration, while DB is most prone to performance bottlenecks. We use read-write separation, vertical sub-library, horizontal sub-library and other schemes.

In the face of high performance and high stability, architecture upgrade needs to be completed in advance as far as possible, otherwise, the system response may slow down or even downtime at any time.

Original link: https://mp.weixin.qq.com/s/IWytJhcxth5ig2qrqdQjvg

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.