Shulou(Shulou.com)06/02 Report--

In the daily operation and maintenance work, nginx services are often used, and the performance bottleneck caused by high concurrency of nginx is often encountered. Today, here is a brief review of the configuration of nginx performance optimization (only based on my actual experience, if there is anything wrong, please point out ~)

1. The optimization here mainly refers to the configuration optimization of nginx. Generally speaking, the following items are useful for optimization in the nginx configuration file:

1) the number of nginx processes, which is recommended to be specified according to the number of cpu, which is generally the same as the number of cpu cores or a multiple of it.

Worker_processes 8

2) assign cpu to each process. In the above example, 8 processes are assigned to 8 cpu. Of course, you can write more than one process, or assign a process to multiple cpu.

Worker_cpu_affinity 00000001 00000010 00000100 00001000 00010000 00100000 01000000 10000000

3) the following instruction refers to the maximum number of file descriptors opened by a nginx process, the theoretical value should be the maximum number of open files in the system (ulimit-n) divided by the number of nginx processes, but the nginx allocation request is not so uniform, so it is best to be consistent with the value of ulimit-n.

Worker_rlimit_nofile 65535

4) use epoll's Istroke O model to deal with asynchronous events efficiently.

Use epoll

5) the maximum number of connections allowed per process, theoretically the maximum number of connections per nginx server is worker_processes*worker_connections.

Worker_connections 65535

6) the http connection timeout is 60s by default. The function is to make the connection between the client and the server remain valid for a set period of time. When a subsequent request to the server occurs, this function avoids establishing or re-establishing the connection. Keep in mind that this parameter can not be set too large! Otherwise, it will cause many invalid http connections to occupy the number of connections in nginx, and eventually nginx crashes!

Keepalive_timeout 60

7) the buffer size of the client request header, which can be set according to the paging size of your system. Generally, the header size of a request will not exceed 1k, but since the paging of the system is generally larger than 1k, it is set to the paging size here. The page size can be obtained with the command getconf PAGESIZE.

Client_header_buffer_size 4k

8) the following parameter specifies the cache for open files. It is not enabled by default. Max specifies the number of caches, which is recommended to be the same as the number of open files. Inactive refers to how long it takes to delete the cache after the file has not been requested.

Open_file_cache max=102400 inactive=20s

9) the following one refers to how often the valid information in the cache is checked.

Open_file_cache_valid 30s

10) the minimum number of times a file is used within the time of the inactive parameter in the open_file_cache directive. If this number is exceeded, the file descriptor is always opened in the cache. As in the example above, if a file is not used once in inactive time, it will be removed.

Open_file_cache_min_uses 1

11) it is good for security to hide the information about the operating system and web server (Nginx) version number in the response header.

Server_tokens off

12) you can make sendfile () work. Sendfile () can copy data (or any two file descriptors) between disk and TCP socket. Pre-sendfile is to request a data buffer in user space before transferring data. The data is then copied from the file to the buffer with read (), and write () writes the buffer data to the network. Sendfile () reads the data from disk to the OS cache immediately. Because this copy is done in the kernel, sendfile () is more efficient than combining read () and write () and turning on and off the discard buffer (more about sendfile).

Sendfile on

13) tell nginx to send all the header files in one packet instead of one after another. That is to say, the data packet will not be sent out immediately, and when the packet is maximum, it will be transmitted at one time, which will help to solve the network congestion.

Tcp_nopush on

14) tell nginx not to cache the data, but to send it one segment at a time-when you need to send data in time, you should set this property to the application so that you can't get an immediate return value when you send a small piece of data information.

Tcp_nodelay on

For example:

Http {server_tokens off; sendfile on; tcp_nopush on; tcp_nodelay on;.}

15) the buffer size of the client request header, which can be set according to the system paging size. Generally, the size of a request header will not exceed 1k, but since the system paging is generally larger than 1k, it is set to the paging size here.

Client_header_buffer_size 4k

The buffer size of the client request header, which can be set according to the system paging size. Generally, the size of a request header will not exceed 1k, but since the system paging is generally larger than 1k, it is set to the paging size here.

The page size can be obtained with the command getconf PAGESIZE.

[root@test-huanqiu ~] # getconf PAGESIZE 4096

However, there are cases where the client_header_buffer_size exceeds 4k, but the client_header_buffer_size value must be set to an integral multiple of the system page size.

16) specify cache for open files. It is not enabled by default. Max specifies the number of caches, which is recommended to be the same as the number of open files. Inactive refers to how long a file has not been requested before deleting the cache.

Open_file_cache max=65535 inactive=60s

17) the minimum number of times a file is used within the time of the inactive parameter in the open_file_cache directive. If this number is exceeded, the file descriptor is always opened in the cache. As in the example above, if a file is not used once in inactive time, it will be removed.

Open_file_cache_min_uses 1

18) specify how often to check for valid information in the cache.

Open_file_cache_valid 80s

Here is a simple nginx configuration file that I use:

[root@dev-huanqiu ~] # cat / usr/local/nginx/conf/nginx.confuser www www;worker_processes 8 http workerboards 00000001 00000010 00000100 00001000 000000 00100000 01000000 http {use epoll; worker_connections 65535; events {include mime.types; default_type application/octet-stream; charset utf-8; server_names_hash_bucket_size 65535 Client_header_buffer_size 2k; large_client_header_buffers 4 4k; client_max_body_size 8m; sendfile on; tcp_nopush on; keepalive_timeout 60; fastcgi_cache_path / usr/local/nginx/fastcgi_cache levels=1:2 keys_zone=TEST:10m inactive=5m; fastcgi_connect_timeout 300; fastcgi_send_timeout 300; fastcgi_read_timeout 300; fastcgi_buffer_size 16k; fastcgi_buffers 16 16k; fastcgi_busy_buffers_size 16k Fastcgi_temp_file_write_size 16k; fastcgi_cache TEST; fastcgi_cache_valid 200 302 1h; fastcgi_cache_valid 301 1d; fastcgi_cache_valid any 1m; fastcgi_cache_min_uses 1; fastcgi_cache_use_stale error timeout invalid_header http_500; open_file_cache max=204800 inactive=20s; open_file_cache_min_uses 1; open_file_cache_valid 30s; tcp_nodelay on; gzip on; gzip_min_length 1k; gzip_buffers 4 16k Gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain application/x-javascript text/css application/xml; gzip_vary on; server {listen 8080; server_name huan.wangshibo.com; index index.php index.htm; root / www/html/; location / status {stub_status on;} location ~. *\. (php | php5)? ${fastcgi_pass 127.0.0.1 gzip_types text/plain application/x-javascript text/css application/xml; gzip_vary on; server 9000 Fastcgi_index index.php; include fcgi.conf;} location ~. *\. (gif | jpg | jpeg | png | bmp | swf | js | css) ${expires 30d;} log_format access'$remote_addr-$remote_user [$time_local] "$request"'$status $body_bytes_sent "$http_referer"$http_user_agent" $http_x_forwarded_for'; access_log / www/log/access.log access;}

2. Several instructions about FastCGI

1) this directive specifies a path, directory structure level, keyword area storage time, and inactive delete time for the FastCGI cache.

Fastcgi_cache_path / usr/local/nginx/fastcgi_cache levels=1:2 keys_zone=TEST:10m inactive=5m

2) specify the timeout for connecting to the backend FastCGI.

Fastcgi_connect_timeout 300

3) the timeout for sending the request to the FastCGI, which refers to the timeout for sending the request to the FastCGI after two handshakes have been completed.

Fastcgi_send_timeout 300

4) the timeout for receiving FastCGI replies, which refers to the timeout for receiving FastCGI replies after two handshakes have been completed.

Fastcgi_read_timeout 300

5) specify the size of the buffer needed to read the first part of the FastCGI reply. Here you can set the buffer size specified by the fastcgi_buffers instruction. The above instruction specifies that it will use a 16k buffer to read the first part of the reply, that is, the response header. In fact, this response header is usually very small (no more than 1k), but if you specify the size of the buffer in the fastcgi_buffers instruction Then it will also allocate a buffer size specified by fastcgi_buffers to cache.

Fastcgi_buffer_size 16k

6) specify how many and how many buffers are needed locally to buffer FastCGI responses. As shown above, if a php script produces a page size of 256k, it will be allocated 16 buffers of 16k to cache. If it is greater than 256k, the portion larger than 256k will be cached in the path specified by fastcgi_temp. Of course, this is an unwise solution for server load, because the processing speed of data in memory is faster than that of hard disk. Usually, the setting of this value should choose a middle value of the page size generated by the php script in your site. For example, if the page size generated by most scripts on your site is 256k, you can set this value to 1616k, or 464k or 64k, but obviously, the latter two are not good settings, because if the resulting page is only 32k, if you use 464k, it will allocate a 64k buffer to cache. If you use 644k, it allocates eight 4k buffers to cache, while if you use 1616k, it allocates two 16k to cache pages, which seems more reasonable.

Fastcgi_buffers 16 16k

7) I don't know what this instruction is for, except that the default value is twice that of fastcgi_buffers.

Fastcgi_busy_buffers_size 32k

8) how many blocks of data will be used when writing to fastcgi_temp_path? the default value is twice that of fastcgi_buffers.

Fastcgi_temp_file_write_size 32k

9) turn on the FastCGI cache and give it a name. Personally, I find it very useful to turn on caching, which can effectively reduce the CPU load and prevent 502 errors. But this cache can cause a lot of problems because it caches dynamic pages. Specific use also needs to be based on their own needs.

Fastcgi_cache TEST

10) specify the caching time for the specified reply code, as in the example above, 200302 replies are cached for 1 hour, 301 responses for 1 day, and others for 1 minute.

Fastcgi_cache_valid 200 302 1h fastcgival any valid 301 1d fastcgival cacheology valid fastchem

11) the minimum number of times during which the fastcgi_cache_path instruction inactive parameter value is cached. As in the example above, if a file is not used once within 5 minutes, the file will be removed.

Fastcgi_cache_min_uses 1

12) not knowing what this parameter does, the guess is to let nginx know which types of caches are useless.

Fastcgi_cache_use_stale error timeout invalid_header http_500

These are the parameters related to FastCGI in nginx

In addition, FastCGI itself has some configurations that need to be optimized. If you use php-fpm to manage FastCGI, you can modify the following values in the configuration file:

1) the number of concurrent requests processed at the same time, that is, it will open up up to 60 child threads to handle concurrent connections.

sixty

2) the maximum number of open files.

65535

3) the maximum number of requests each process can execute before resetting.

65535

Third, on the optimization of kernel parameters, in the / etc/sysctl.conf file

1) the number of timewait. Default is 180000. (Deven: so if you want to lower the timewait, you have to reduce the tcp_max_tw_ buckets.)

Net.ipv4.tcp_max_tw_buckets = 6000

2) the range of ports that the system is allowed to open.

Net.ipv4.ip_local_port_range = 1024 65000

3) enable the TIME-WAIT status sockets fast recycling feature; used to quickly reduce the number of TCP connections in the TIME-WAIT state. 1 means enabled; 0 means off. However, it is important to note that this option is generally not recommended, because in NAT (Network Address Translation) networks, it will lead to a large number of TCP connection establishment errors, resulting in website access failures.

Net.ipv4.tcp_tw_recycle = 0

In fact, when the net.ipv4.tcp_tw_recycle function is turned on, it will have no effect until net.ipv4.tcp_timestamps (which is enabled by default in general systems) is enabled.

When tcp_tw_recycle is turned on (tcp_timestamps is turned on at the same time, the effect of quickly recycling socket is achieved), it is a disaster for Client, which is located behind the NAT device!

It can cause the Server of the Client connection to the back of the NAT device to be unstable (some Client can connect to server and some Client cannot connect to server).

In other words, tcp_tw_recycle is designed for the internal network (the network environment can be controlled by itself-there is no NAT), and it is not suitable for use in the public network environment.

Generally speaking, socket in TIME_WAIT state is reclaimed because it "cannot actively connect to the remote end" because there are no ports available, and memory should not be reclaimed (not necessary).

That is: the demand is the requirement of Client, will Server have the problem of "insufficient ports"?

Unless it is a front-end machine, you need to connect a large number of back-end services, that is, to play the role of Client.

The correct solution to this is always:

Net.ipv4.ip_local_port_range = 9000 6553 # default value range is smaller net.ipv4.tcp_max_tw_buckets = 10000 # default value is smaller, and net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 can be adjusted appropriately.

4) enable the reuse feature to allow sockets in the TIME-WAIT state to be reused for new TCP connections. It is safe to enable this function, generally do not change it!

Net.ipv4.tcp_tw_reuse = 1

5) enable SYN Cookies, and enable cookies to handle when SYN waiting queue overflow occurs.

Net.ipv4.tcp_syncookies = 1

6) the backlog of listen function in web application limits the net.core.somaxconn of kernel parameters to 128by default, while the NGX_LISTEN_BACKLOG defined by nginx defaults to 511.Therefore, it is necessary to adjust this value.

Net.core.somaxconn = 262144

7) the maximum number of packets allowed to be sent to the queue when each network interface receives packets faster than the kernel processes them.

Net.core.netdev_max_backlog = 262144

8) the maximum number of TCP sockets in the system that are not associated to any of the user file handles. If this number is exceeded, the orphan connection will be immediately reset and a warning message will be printed. This limit is only to prevent simple DoS attacks, do not rely too much on it or artificially reduce this value, but should increase this value (if you increase memory).

Net.ipv4.tcp_max_orphans = 262144

9) the maximum value of recorded connection requests for which the client acknowledgement has not been received. For systems with 128 megabytes of memory, the default is 1024, and for systems with small memory, it is 128.

Net.ipv4.tcp_max_syn_backlog = 262144

10) timestamps can avoid the winding of serial numbers. A 1Gbps link is sure to encounter a sequence number that has been used before. Timestamps allow the kernel to accept such "abnormal" packets.

Net.ipv4.tcp_timestamps = 1

In order to improve the performance, many servers turn on the net.ipv4.tcp_tw_recycle option. In the NAT network environment, it is easy to cause some connect failures when visiting the website.

Personal advice:

Turn off the net.ipv4.tcp_tw_recycle option instead of net.ipv4.tcp_timestamps

Because when net.ipv4.tcp_timestamps is off, turning on net.ipv4.tcp_tw_recycle does not work; while net.ipv4.tcp_timestamps can be turned on and works independently.

11) to open a peer-to-peer connection, the kernel needs to send a SYN with an ACK that responds to the previous SYN. It is the second handshake in the so-called three-way handshake. This setting determines the number of SYN+ACK packets sent by the kernel before the connection is abandoned.

Net.ipv4.tcp_synack_retries = 1

12) the number of SYN packets sent before the kernel gives up establishing the connection.

Net.ipv4.tcp_syn_retries = 1

13) if the socket is closed by the local request, this parameter determines how long it remains in the FIN-WAIT-2 state. The peer can make an error and never close the connection, or even crash unexpectedly. The default value is 60 seconds. 2.2 the usual value of the kernel is 180s, you can press this setting, but keep in mind that even if your machine is a light WEB server, there is a risk of memory overflow due to a large number of dead sockets. FIN-WAIT- 2 is less dangerous than FIN-WAIT-1 because it can only eat up to 1.5K of memory, but they last longer.

Net.ipv4.tcp_fin_timeout = 30

14) the frequency at which TCP sends keepalive messages when keepalive is activated. The default is 2 hours.

Net.ipv4.tcp_keepalive_time = 30

Here is a standard configuration of kernel parameters that I often use.

[root@dev-huanqiu ~] # cat / etc/sysctl.confnet.ipv4.ip_forward = 0net.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.accept_source_route = 0kernel.sysrq = 0kernel.core_uses_pid = 1net.ipv4.tcp_syncookies = 1 / / these four lines of red content Generally, when a large number of TIME_WAIT is found, the solution kernel.msgmnb = 65536kernel.msgmax = 65536kernel.shmmax = 68719476736kernel.shmall = 4294967296net.ipv4.tcp_max_tw_buckets = 6000net.ipv4.tcp_sack = 1net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_rmem = 4096 87380 4194304net.ipv4.tcp_wmem = 4096 16384 4194304net.core.wmem_default = 8388608net.core.rmem_default = 8388608net.core.rmem_max = 16777216net.core.wmem_max = 16777216net.core.netdev_ Max_backlog = 262144net.core.somaxconn = 262144net.ipv4.tcp_max_orphans = 3276800net.ipv4.tcp_max_syn_backlog = 262144net.ipv4.tcp_timestamps = 1 / / when net.ipv4.tcp_tw_recycle is set to 1 It is best to add net.ipv4.tcp_synack_retries = 1net.ipv4.tcp_syn_retries = 1net.ipv4.tcp_tw_recycle = 1 / / enable this feature to reduce TIME-WAIT status, but opening it in NAT network mode may lead to tcp connection errors, so be careful. Net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_mem = 94500000 915000000 927000000net.ipv4.tcp_fin_timeout = 30net.ipv4.tcp_keepalive_time = 30net.ipv4.ip_local_port_range = 1024 65000net.ipv4.ip_conntrack_max = 6553500

-- remember a minor accident

When the net.ipv4.tcp_tw_recycle = 1 function is turned on, it does reduce the TIME-WAIT status, which I used to turn on.

But also because of this parameter, I stepped on a pit once:

One of the company's news release CMS background system, using haproxy+keepalived proxy architecture, the back-end real server server extranet ip all removed.

Phenomenon: during the peak period of sending messages in the morning, there is an access failure in the CMS backend, which will take effect immediately after restarting the php service, but after it lasts for a period of time, the access fails again.

Check the nginx and php logs did not find anything, and then google for a while, found that this parameter net.ipv4.tcp_tw_recycle is the trick!

This network architecture is NAT mode for the back-end realserver. When this parameter is turned on, it will lead to a large number of TCP connection establishment errors, resulting in website access failures.

Finally, set net.ipv4.tcp_tw_recycle to 0. After turning off this function, the background access immediately returns to normal.

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.