Shulou(Shulou.com)06/02 Report--

This article introduces the knowledge of "online Mall system High concurrency Optimization method tutorial". In the operation of actual cases, many people will encounter such a dilemma. Next, let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Brief overview of the project

The project is a single mall architecture project based on SSM architecture, in which there is a big module. The following is a brief architecture deployment diagram of the current online environment.

Give a general description:

The project is SSM architecture.

Server category: 1 load balancing server (F5), 3 application servers, 1 timer server, 1 Redis server, 1 picture server and 1 MySQL master-slave server based on Pass architecture (Microsoft Cloud).

Call logic: the figure above is a brief call logic.

What is a single architecture project?

From the perspective of architectural development, the software project has gone through the following stages of development:

Single architecture: can be understood as the traditional front-end and front-end unseparated architecture.

Vertical architecture: can be understood as the front and back end separate architecture.

SOA architecture: it can be understood as a service-oriented architecture based on service category, business flow, service dependency, etc., such as the previous single architecture ERP project, divided into order service, procurement service, material service and sales service, etc.

Micro services: can be understood as small projects, such as the previous ERP large projects, divided into order services (order items), procurement services (procurement items), material services (material items) and sales services (sales items), and calls between services.

Online problems caused by this SSM project

① when the second kill, CPU soars

The daily second kill of the system is divided into three time ends: 10:00, 13:00 and 20:00. The following is a brief page of the second kill:

② single application server CPU

Number of requests for a single application server in ③

Number of ④ rdis connections (info clients)

This unsaved screenshot is about 600.

Connected_clients:600

⑤ MySQL request screenshot

Investigation process and analysis

Investigation train of thought

According to the service deployment and project architecture, troubleshoot from the following aspects:

Application server: check memory, CPU, number of requests, etc.

File image server: check memory, CPU, number of requests, etc.

Timer server: check memory, CPU, number of requests, etc.

Redis server: check memory, CPU, number of connections, etc.

DB server: check memory, CPU, number of connections, etc.

Investigation process

Within 30 minutes of the second kill:

① application server CPU soars, memory soars, the root cause of CPU and memory surge is that the number of requests is too high, a single application server reached more than 3000.

The ② Redis request timed out, as shown below:

The ③ JDBC connection timed out, as shown below:

④ checks through GC and finds that FullGC has occurred 152 times in 24 hours, as shown below:

⑤ looked at the stack and found some thread blocking and deadlocks.

Jstat-l pid, you can also use VisualVM analysis:

⑥ found that more than 2000 threads requested invalid resources, as shown in the following figure:

Analysis of the main factors causing the abnormality of the system

The main factors causing the abnormality of the system are as follows:

In the second kill, the number of requests is too high, resulting in the use of server load is too high.

The Redis connection pool is full and the connection cannot be obtained, connot get a connection from thread pool.

The JDBC connection pool is full, unable to get a connection and timed out.

There is large object code, such as constantly adding objects to the List collection, failing to collect objects in time leads to memory increase, and Full GC occurs frequently.

Tomcat concurrency parameters, JVM optimization parameters, Jedis configuration parameters, JDBC configuration parameters are unreasonable.

There is no peak cutting and current limit for the requested quantity.

Resource connections are not released in time, such as Redis connections and JDBC connections.

Final solution

① increases the use of services, peak cutting and shunting.

Since the project does not add MQ, we can only use hard load and increase server horizontal expansion to achieve traffic peaking and traffic diversion:

② optimizes JVM parameters. The following are the optimized parameters:

JAVA_OPTS= "- server-Xmx9g-Xms9g-Xmn3g-Xss500k-XX:+DisableExplicitGC-XX:+DisableExplicitGC-XX:MaxMetaspaceSize=2048m-XX:+UseConcMarkSweepGC-XX:+CMSParallelRemarkEnabled-XX:LargePageSizeInBytes=128m-XX:+UseFastAccessorMethods-XX:+UseCMSInitiatingOccupancyOnly-XX:CMSInitiatingOccupancyFraction=70-Dfile.encoding=UTF8-Duser.timezone=GMT+08"

With regard to the optimization of this JVM parameter, what the JVM theory is, what the official advice is, and what the actual combat is, will be analyzed in the next article.

③ optimizes Tomcat concurrency related parameters

Mainly in two aspects:

Change the bio protocol to nio2.

Set the relevant parameters reasonably according to the server configuration, business scenario, business traffic and so on, and try to achieve the best.

The optimization of Tomcat-related parameters will be analyzed in the following article.

Parameter optimization of ④ Redis and JDBC

It is not listed here because of the security issues involved.

⑤ code optimization

The code is optimized as follows:

Optimize the size of the object.

Optimize objects and connection resources that are not released in time.

⑥ solves the problem of more than 1000 threads requesting invalid resources:

Increase the cache in conf/context.xml

Final optimization result

After several days of observation, the system is stable.

Basic monitoring, as shown below:

GC, as shown below:

Sampler CPU and memory:

This is the end of the "online mall system high concurrency optimization method tutorial". Thank you for your reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.