Shulou(Shulou.com)06/02 Report--

Author: wx formyz

My advice

At present, there are still some people (including some programmers) who think that there is no need for a professional system administrator (commonly known as operation and maintenance dog) to use a cloud host, search the Internet, and configure the installation documents. Of course, there are hints in the promotion of cloud service providers (buy a cloud host, be stable, and throw the data up once and for all). Is that really the case? If your application has little traffic and few people visit it a day, you really don't have to spend money to hire a full-time system administrator; if you rely on the Internet to support a group of people, and want more users to access, and the above awareness, I can only hehe …

Environments that are not deployed by system administrators

An application, all on a public cloud. It is composed of load balance, four web applications, shared data disk (program code sharing), database (master-slave) and other parts. Structurally, well, there's nothing wrong with it. So, for a long time, no one came to us for support, and we didn't know these applications existed.

Autumn is coming, and the weather in Didu is good. I guess everyone must be in the same mood as the weather. Recently, however, some people in the technical support qq group have been calling that the load of all four servers in the project is skyrocketing. Load can reach hundreds from 9 to 11:00 in the evening. It was said that the relevant personnel had been investigating for several days, but to no avail (I snickered alone).

If the application is nginx + php + mysql, the possible bottlenecks and areas that can be adjusted roughly include: system configuration, php configuration, database configuration (cloud service provider's load balancer is not adjustable). Let's cut the gossip and rush it so fast. Let's take a look at the operation first.

My god, running so high is not dead yet, so give me a start. In addition to the high cpu load, memory is also basically exhausted. According to past experience, there may be a problem with the setting of the system parameters (the default systcl.conf is not set), so arrange for my little brother to refer to it from another server, set it, and execute sysctl-p to make it effective. Follow-up observation at the peak of the visit, the result is not good, it seems that you have to do it yourself.

Log in to the system at the selected time, that is, one hour before the peak of access.

First, let's see what dries up the memory. Ps looks at the process and finds that there is a lot of php going on. It is initially suspected that after the user has requested data, in order to effectively shut down the process and release resources, restart the php service with permission. In a moment, the process ran out of memory again, something is wrong!

Repeat the following command to count the php process:

Ps auxww | grep php | grep-v grep | wc-l

Ps auxww | grep php | grep-v grep | wc-l

The number of processes remains the same, at 601. A process takes up several megabytes of memory, 600 processes, with a minimum consumption of gigabytes of memory, no wonder the load is not high.

Open the configuration file php-fpm.conf and see the problem at a glance

Process management is incorrectly set to static (static), and the maximum child process is 600. once php is started, one main process plus 600 child processes will be started, whether necessary or not. Parameters related to dynamic management, such as maximum start process, maximum number of idle processes, and so on, are invalid after the line of configuration file php-fpm.conf maximum child process. After correcting this problem, the time is almost up to the peak of visits. Through manual tracking and monitoring alarm, basically there is a great improvement, the peak load load is less than 50.

Although the performance has been improved by modifying the php parameter setting, I am still not satisfied with the result. I'd like to see if there is anything I can adjust. So, the idea came to the problem of disk io.

Four servers share a cloud nas hard disk and only one copy of the php code written by the programmer is saved. If the performance of io is poor, it will seriously affect the performance of the whole application.

Use the mount instruction to check the mounting of nfs, mainly the mounting parameters. The results are as follows:

I used the tcp protocol, while in previous practice, I usually used the udp protocol (vers=3) for hooking. Considering the disk performance provided by cloud service providers, tcp may not be better than udp. So negotiate with others, without affecting the performance access, first modify the connection mode of one server to nfs, and then modify other servers after further performance improvement, and finally leave one without change, in order to observe the comparison effect.

Turn off the service, switch out of the mount point directory, uninstall nfs, and reattach nfs with the following instructions:

/ usr/bin/mount-t nfs-o nolock,vers=3 6e46868719-pgn67.cn-qingdao.nas.aliyuncs.com:/ / data

/ usr/bin/mount-t nfs-o nolock,vers=3 6e46868719-pgn67.cn-qingdao.nas.aliyuncs.com:/ / data

Then start php and other related services, the peak period, the effect is very obvious, load value is reduced to less than 5.

After several days of observation, by comparison, the nfs mounting method of CVM also has a great impact on performance.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.