Shulou(Shulou.com)06/02 Report--

I. Some of the problems encountered

I remember that when I did performance testing in 2008, I newly purchased 7 Lenovo 4-core 4G servers for performance testing.

At that time, resources were tight, and these seven servers were equipped with dual systems (Win2003/CentOS5) for testing machines (pressure test agents) when idle.

At that time, a series of tests were done for Nginx, and what impressed me deeply was: on these machines, the pressure test of Nginx status page.

The best QPS for short connections is about 40,000, and the highest QPS for long connections is about 130,000.

About three years later, those lenovo servers were no longer worthy of anyone's attention and could only be used as chickens.

However, a casual test found that the best QPS for short connections was not much higher for any server. Moreover, the resources of the tester were not exhausted, the resources of the server under test were also exhausted, and there were no bottlenecks in the network.

Server resource usage was low, but the response just wasn't fast enough.

Finally, we found the bottleneck at the listening portal! Can I improve the performance of listening entries? Is it possible to multiplex ports? Finally we found SO_REPORT.

SO_REUSEPORT supports multiple processes or threads bound to the same port, improving the performance of server programs.

II. Solutions

test environment

Dell PowerEdge M620 Intel(R)Xeon(R)CPU E5-2620v2@2.10GHz

Linux3.16.0-4-amd64#1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux

Ethernet controller:Broadcom Corporation NetXtreme II BCM5781010Gigabit Ethernet(rev10)

View compilation parameters

Nginx configuration is as follows:

Note that there is a reuse_port parameter

user www-data;worker_processes auto;pid/run/nginx.pid;events{useepoll; multi_accept on; reuse_port on; worker_connections 1048576;}dso{#Dynamic Load Function Module/usr/share/nginx/modulesload ngx_http_memcached_module.so;load ngx_http_limit_conn_module.so;load ngx_http_empty_gif_module.so;load ngx_http_scgi_module.so; load ngx_http_upstream_session_sticky_module.so;load ngx_http_user_agent_module.so; load ngx_http_reference_module.so; load ngx_http_upstream_least_conn_module.so; load ngx_http_uwsgi_module.so; load ngx_http_reqstat_module.so;load ngx_http_browser_module.so;load ngx_http_limit_req_module.so;load ngx_http_split_clients_module.so;load ngx_http_upstream_ip_hash_module.so;}http{include /etc/nginx/mime.types;default_type text/plain;access_log off;sendfile on;tcp_nopush on;tcp_nodelay on;server_tokens off;keepalive_timeout 120;server_names_hash_bucket_size512;server_name_in_redirect off;fastcgi_connect_timeout3s;fastcgi_send_timeout3s;fastcgi_read_timeout3s;fastcgi_buffer_size128k;fastcgi_buffers8128k;fastcgi_busy_buffers_size256k;fastcgi_temp_file_write_size256k;variables_hash_max_size 1024;set_real_ip_from10.0.0.0/8;set_real_ip_from172.28.0.0/16;set_real_ip_from192.168.0.0/16;real_ip_headerX-Forwarded-For;gzip off;gzip_disable"msie6";gzip_min_length2k;gzip_buffers1664k;gzip_http_version1.1;gzip_comp_level6;gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;gzip_vary on;ssl_protocols TLSv1 TLSv1.1TLSv1.2;# Dropping SSLv3, ref: POODLEssl_prefer_server_ciphers on;access_log/var/log/nginx/access.log;error_log/var/log/nginx/error.log;server{listen 80backlog=65535;charset utf-8;location/{#Print Engine status page stub_status on;#Open status page, dependent on http_stub_status_module access_log off;#Access process does not log}location~^(.*)\/\. (svn|hg| bzr| cvs)\/{#block these directories deny all;access_log off;log_not_found off;}location~/\.| cvs)\/{# 屏蔽这些目录deny all;access_log off;log_not_found off;}location~/\. {#shield. directory or file at the beginning, such as.htaccess .bash_historydeny all;access_log off;log_not_found off;}location/do_not_delete.html{access_log off;empty_gif;}}}

Pressure test reuse_port

Tengine already supports reuse_port. When you turn on reuse_port, you will find that there are many processes listening on port 80 at the same time:

After pressurization, you will find that server performance can be drained by you:

Compared with the effect of testing reuse_port, my friends were shocked (short connection QPS exceeded 240,000)!

When the truth comes out, what are you waiting for?

find out

TCP: Possible SYN flooding on port 80 A lot of mistakes.

This reduces concurrency to 60,000 net.core.somaxconn = 65535.

After closing reuse_port, let's look at perf top:

Then turn on reuse_port and compare perf top:

At this point, zoom in on the back_log that Nginx listens to see how resources are used:

Let's look at the queue situation at some time (there are more than 10,000 queues):

Then we challenge 300,000 concurrent (MTT is the average response time (ms)):

TCP: Possible SYN flooding on port 80. However, a small number of connection timeouts occurred:

So far the test is over, open reuse_port can really make performance up to 3 times, why not try.

The above is all the content of this article, I hope to help everyone's study, but also hope that everyone a lot of support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.