Shulou(Shulou.com)06/03 Report--

Background introduction

Zaimi is an Internet enterprise focusing on campus e-commerce, currently specializing in campus supermarket O2O. The company was founded in November 2014. in just over a year, the company has gone through four rounds of financing, covering nearly 200 cities, more than 1000 colleges and universities, more than 10000 dormitory buildings, 200000 daily orders and 500000 peak orders.

A preliminary understanding of the structure

Can such a system cope with the rapid business development in the future? Will performance problems become a bottleneck in the growing trading volume? Can the system withstand large-scale concurrent access during peak access periods? Performance optimization has become the most important work at this time, so special engineers are arranged to carry out performance testing and performance optimization, sort out the system status from the architecture, code, database, operation and maintenance levels, find the system bottleneck, and optimize it.

Xiaoqiang comments: this kind of architecture is the first time to know the architecture, and the general system is gradually evolved on this architecture. It is very important for test engineers to master the basic architecture knowledge.

Performance testing

The characteristic of campus snack shopping is that it reaches a peak around 10:00 in the evening, and the trading volume in one hour before and after that accounts for about half of the trading volume all day, that is to say, if you want to design a system of 1 million daily orders, in fact, the trading pressure is 500000 orders per hour.

According to the 2008 rule, the peak number of calls per second was estimated to be 556. Based on this, the back-end API call frequency was analyzed according to the Nginx log, and the request for the top 20% of the API call ratio was calculated to construct a test scenario.

When performing the performance test, we use Jmeter as the performance testing tool and take advantage of the system resource monitoring provided by the cloud service as the basis.

Xiaoqiang comments: many people have been obsessed with how to calculate the number of concurrent issues. in fact, there are a lot of algorithms. In the Xiaoqiang performance testing training class, I talked about about four algorithms. Personally, I think there is no question of whether it is accurate at all. The key is in your application scenario. Where in the world is absolutely accurate?

Architecture optimization

The performance test results are not optimistic, although the system previously used distributed cache to cache hot data, but it is relatively random, which data need to be cached and how to set the failure strategy have not been carefully analyzed and designed. After the performance test, we decided to standardize the use of cache, cache all kinds of frequently read data as much as possible, and deploy the Redis server for cluster and master-slave replication.

Xiaoqiang comments: the effect of caching is obvious, generally for a large number of data queries we have to first consider the cache optimization. Of course, we're talking about back-end caching, and front-end caching also needs to be considered.

In addition, third-party CDN services are also used to accelerate static file access. Product pictures, JavaScript files, CSS files and so on are accelerated through CDN. At the same time, the front-end cache of static files is provided through Nginx reverse proxy server.

Xiaoqiang comments: these are biased towards the front-end performance optimization issues, interested can see my front-end performance test video.

The performance test found that the main bottleneck of the system is the database. Although Redis is used to cache the hot data, the database still shows the system overload after the concurrency reaches a certain degree. So the master and slave of the database are separated.

Sql statement optimization

In the process of performance testing, it is found that because the main efforts are focused on how to quickly implement the business, a large number of database query statements are written more casually, and the index design is very unreasonable. Combined with the slow.log analysis of the Mysql database in the performance test, locate the slow query SQL to append index, and then optimize the SQL using the interpreted execution plan explain.

Xiaoqiang comments: any system in the sql statement will have more or less problems, generally our method is slow query monitoring > top N sentence analysis > optimization and improvement. The train of thought is basically like this, the specific approach is different, flexible response.

Database connection pool optimization

When doing performance testing, it is found that there are more serious performance problems in some cases. In the case of high concurrency, the application is inaccessible under pressure for a long time.

Looking for information on the Internet, I found that many people also encountered the "APPARENT DEADLOCK" problem of C3P0.

After switching C3P0 to domestic database connection pool Druid, the situation improved obviously, and similar problems never appeared again.

Xiaoqiang comments: C3P0 does have some small bug. Later, we also used Ali open source Druid, which is still good at present, so we can give it a try.

H5 response compression optimization

Enable Nginx gzip compression to reduce the size of App response packets and improve response performance

Xiaoqiang comments: front-end performance optimization, whether app, H6 or web are the same, many of our children's shoes are too rigid to achieve one-size-fits-all.

Hot and cold separation of order data

With the continuous development of the business, the data in the order table will be more and more. According to our current daily order estimate of 200000 orders, the monthly order is 6 million, and the annual order is 72 million, and the daily order is still increasing. Before long, the amount of data will exceed the limit of MySQL. At the beginning, we considered the scheme of using distributed database, split the order table horizontally, use the order number for hash, and split the order data into multiple tables. After further analysis, it is found that the order data is obviously hot and cold uneven, that is, the orders just created are hot data, and different applications access and modify these orders in various ways. After a period of time, especially after the completion of the order, the frequency of order access decreases sharply, and there is only one kind of operation: order query. Therefore, we consider to adopt the strategy of hot and cold data separation to control the total amount of data in the hot database and ensure that the amount of data in the order table is always maintained within an acceptable range, so as to provide stable data access performance.

Xiaoqiang comments: we always feel that we have to use advanced technology to appear fierce, in fact, this is pretending to force. When we were in Sina, we did a lot of sub-database and sub-table on the database, but the final performance improvement was not obvious. I have always stressed that technology serves the business, only by defining the characteristics of the business, according to the business to use reasonable technology can be greatly improved, otherwise it will be counterproductive.

Summary

Performance problems are real problems, and solutions should be aimed at specific problems one by one. Understand the current situation of the system through performance testing, find specific problems through bottleneck analysis, find solutions for specific problems, achieve solutions and then carry out performance testing, the whole performance optimization forms a closed loop, and the system can be continuously optimized.

Xiaoqiang comments: performance optimization is a continuous process, no one can optimize the road in one step, and no one can optimize to the extreme. He needs you to have a perfect knowledge system and understand all aspects. It can't be accomplished by a LoadRunner or jmeter. These are only a small part of the performance. Let's refuel together.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.