Shulou(Shulou.com)06/01 Report--

This article will give you a detailed explanation on how to optimize Linux Netfilter. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

If you are serving a high-traffic Web/DNS server and have recently lost the server PING and not all HTTP requests have been successful. You can start checking the Syslog. And if you see something similar to the following, the following guidelines will help you tune the Linux server to handle the traffic load correctly.

Mar 22 21:25:55 localhost kernel: nf_conntrack: table full, dropping packet.

Mar 22 21:26:00 localhost kernel: printk: 11 messages suppressed.

Mar 22 21:26:00 localhost kernel: nf_conntrack: table full, dropping packet.

Mar 22 21:26:05 localhost kernel: printk: 16 messages suppressed.

Status View

Buckets hash table size, the number of connections recorded by max maximum

Sudo dmesg | grep conntrack

[8.782060] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)

Hash table usage

Grep conntrack / proc/slabinfo

Nf_conntrack_1 102 102 320 51 4: tunables 0 0 0: slabdata 2 20

Number of connections currently tracked

Sudo sysctl net.netfilter.nf_conntrack_count

Track connection details

# centos

Cat / proc/net/nf_conntrack

# ubuntu, you may need to install conntrack tools, yum install-y conntrack or apt-getinstall-y conntrack

Conntrack-L

Maximum number of connection tracks

In order to complete the task, NAT-server (which generally refers to iptables) needs to record all connections through it. Whether it's "ping" or someone's "ICQ", NAT-server records it in a special table and tracks all conversations. When the session is closed, the related records are deleted from the connection tracking table. The size of this record table is fixed, so if the traffic through the server is large, but the table is too small, NAT-server will start dropping packets and interrupting the session. In order to avoid such trouble, it is necessary to increase the size of the join tracking table appropriately.

The maximum number of connection traces defaults to * nf_conntrack_buckets * 4 tracks. You can view the current value by using the following command:

Sysctl net.netfilter.nf_conntrack_buckets

Sysctl net.netfilter.nf_conntrack_max

CONNTRACK_MAX default calculation formula

CONNTRACK_MAX = number of memory * 1024 "1024" 1024 Universe 16384 / (ARCH/32)

Where ARCH is the CPU schema and the value is 32 or 64.

For example, a machine with 64-bit 8G memory: (8 * 1024 ^ 3) / 16384 / (64thumb 32) = 262144

Temporary adjustment

The temporary adjustment is temporary, and the good configuration value of the restart node will be lost.

Sysctl-w net.netfilter.nf_conntrack_max=1048576

Sysctl-w net.nf_conntrack_max=1048576

Permanent adjustment

To make its configuration permanent after a reboot, you need to add these values to the sysctl.conf

Echo 'net.netfilter.nf_conntrack_max' = 1048576 > > / etc/sysctl.conf

Echo 'net.nf_conntrack_max = 1048576' > > / etc/sysctl.conf

Sysctl-p

If the RAM in the server is less than 1 GB, it is recommended that you do not set too large a value.

Hash table (hash-table)

The hash table size is read-only and cannot be set in the / etc/sysctl.conf file. In 64-bit Linux systems, 4G memory defaults 16384p 8G memory defaults 65536p 16g double, and so on.

The impact of expanding the hash table

It is mainly due to the increased use of memory, and the 32-bit system is also concerned about whether the address space in the kernel state is enough.

The hash table of netfilter stores kernel memory space, which cannot be swap. In order to be compatible with 32 bits, the default value of the operating system is often conservative.

The virtual address space of 32-bit system is up to 4G, of which the kernel state is up to 1G, and only the front 896m can be used.

Allocating too much address space to netfilter may result in underallocation of other kernel processes. One trace record is about 300 bytes, so the current year's nf_conntrack_max defaults to 65535, accounting for more than 20 MB.

The virtual address space of the 64-bit system is 256 TB, which can be used by half of the kernel state, and you only need to care about the use of physical memory.

A formula for calculating memory usage

Size_of_mem_used_by_conntrack (in bytes) = CONNTRACK_MAX * sizeof (struct ip_conntrack) + HASHSIZE * sizeof (struct list_head)

Sizeof (struct ip_conntrack) varies with different architectures, kernel versions, and compilation options. Here it is calculated as 352 bytes.

Sizeof (struct list_head) = 2 * size_of_a_pointer (pointer size for 32-bit systems is 4 bytes, 64 bits is 8 bytes)

For a 64-bit system with 8 GB memory, the default CONNTRACK_MAX is 262144 when HASHSIZE is 65536: 262144 * 352 + 65536 * 8 = 92798976 (88.5 MB)

Internet companies' servers are usually not so tight on memory, so you can let go:

CONNTRACK_MAX is 1048576 and HASHSIZE is 262144. Memory usage: 1048576 * 352 + 262144 * 8 = 371195904 (354 MB)

Hash table resizing

You need to modify it through the kernel module:

Temporary effect:

Echo 262144 > / sys/module/nf_conntrack/parameters/hashsize

Take effect permanently

Add the following to the file: / etc/modprobe.d/iptables.conf (if not, create a new one)

Echo 'options nf_conntrack hashsize=262144' > > / etc/modprobe.d/iptables.conf

Reduce timeout

NAT-server only tracks active sessions that pass through it. If a session is idle and inactive for a long time, it will be closed because of overvalue. When the session is closed, the information about it is deleted so that the connection tracking table does not overflow.

However, if the default value of the timeout is large, and when the traffic is large, even if the nf_conntrack_max is extended to the limit, the trace table still has the risk of overflow. To do this, the connection tracking timeout must be set correctly on NAT-server.

You can view the default values by executing the following command:

Sysctl-a | grep conntrack | grep timeout

Ubuntu 16.04

Net.netfilter.nf_conntrack_generic_timeout = 600,

Net.netfilter.nf_conntrack_icmp_timeout = 30

Net.netfilter.nf_conntrack_tcp_timeout_close = 10

Net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60

Net.netfilter.nf_conntrack_tcp_timeout_established = 432000

Net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120

Net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30

Net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300

Net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60

Net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120

Net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120

Net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300

Net.netfilter.nf_conntrack_udp_timeout = 30

Net.netfilter.nf_conntrack_udp_timeout_stream = 180

Centos 7.8

Net.netfilter.nf_conntrack_dccp_timeout_closereq = 64

Net.netfilter.nf_conntrack_dccp_timeout_closing = 64

Net.netfilter.nf_conntrack_dccp_timeout_open = 43200

Net.netfilter.nf_conntrack_dccp_timeout_partopen = 480

Net.netfilter.nf_conntrack_dccp_timeout_request = 240,

Net.netfilter.nf_conntrack_dccp_timeout_respond = 480

Net.netfilter.nf_conntrack_dccp_timeout_timewait = 240,

Net.netfilter.nf_conntrack_events_retry_timeout = 15

Net.netfilter.nf_conntrack_generic_timeout = 600,

Net.netfilter.nf_conntrack_icmp_timeout = 30

Net.netfilter.nf_conntrack_sctp_timeout_closed = 10

Net.netfilter.nf_conntrack_sctp_timeout_cookie_echoed = 3

Net.netfilter.nf_conntrack_sctp_timeout_cookie_wait = 3

Net.netfilter.nf_conntrack_sctp_timeout_established = 432000

Net.netfilter.nf_conntrack_sctp_timeout_heartbeat_acked = 210,

Net.netfilter.nf_conntrack_sctp_timeout_heartbeat_sent = 30

Net.netfilter.nf_conntrack_sctp_timeout_shutdown_ack_sent = 3

Net.netfilter.nf_conntrack_sctp_timeout_shutdown_recd = 0

Net.netfilter.nf_conntrack_sctp_timeout_shutdown_sent = 0

Net.netfilter.nf_conntrack_tcp_timeout_close = 10

Net.netfilter.nf_conntrack_tcp_timeout_close_wait = 3600

Net.netfilter.nf_conntrack_tcp_timeout_established = 86400

Net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120

Net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30

Net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300

Net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60

Net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120

Net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120

Net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300

Net.netfilter.nf_conntrack_udp_timeout = 30

Net.netfilter.nf_conntrack_udp_timeout_stream = 180

The above are timeout values in seconds.

For servers connected to the public network, consider adjusting the following parameters to reduce the harm of DDoS:

Net.netfilter.nf_conntrack_tcp_timeout_established: default 432000 (5 days)

The scenario corresponding to this value is "after the connection is established, the two parties do not send the package until 5 days later."

However, the default keep-alive timeout is only 2 hours and 11 minutes (net.ipv4.tcp_keepalive_time + net.ipv4.tcp_keepalive_intvl * net.ipv4.tcp_keepalive_probes). Because the timeout is off and socket is not sent, conntrack cannot know the change of status according to the identification of the packet header, and the record will remain in ESTABLISHED status until the countdown ends 5 days later.

The best target for an empty connection attack. The attacker changes the source address of the IP header to random IP, turns off socket after shaking hands, and fills your hash table with a request from a machine.

Net.netfilter.nf_conntrack_tcp_timeout_syn_recv: default is 60

Similarly, an ACK that deliberately does not shake hands can be used. But this timeout is not so exaggerated, and the system also has a syn cookie mechanism to mitigate syn flood attacks.

Other noteworthy parameters:

Net.netfilter.nf_conntrack_tcp_timeout_syn_sent: default 120

Is the connect timeout of your program that long?

Net.netfilter.nf_conntrack_tcp_timeout_fin_wait: default 120

Net.ipv4.tcp_fin_timeout defaults to 60 seconds and is usually set to a smaller value with reference to BSD and macOS. It doesn't have to be this big.

Net.netfilter.nf_conntrack_icmp_timeout: default is 30

Where does the ping wait 30 seconds to time out?

These are reasonable, less than or equal to the extreme cases that may be encountered, but if you do not want the records of semi-closed connections to continue to occupy the valuable hash table, it seems no problem to clear them early:

Net.netfilter.nf_conntrack_tcp_timeout_time_wait: default 120

MSL in Linux is dead for 60 seconds (instead of slapping the head for 120 seconds in the TCP standard). TIME_WAIT has to wait for 2MSL, which is a reasonable value.

But now there is PAWS (net.ipv4.tcp_timestamps) by default, and there will be no situation in which the misguided message that was feared during standard-setting happens to contaminate the data of the new connection with the same serial number. Internet companies basically open net.ipv4.tcp_tw_reuse, since semi-connections do not stay so long, records do not seem to need to be kept for so long.

Net.netfilter.nf_conntrack_tcp_timeout_close_wait: default is 60

The CLOSE_WAIT status is to allow the passive shutdown party to finish transmitting the data that should be transmitted. If the program is not well written and uncaught exceptions are thrown here, you may not be able to send a FIN and stop here all the time.

Net.netfilter.nf_conntrack_tcp_timeout_last_ack: default is 30

If you can't receive the opposite ACK or RST after sending FIN passively, it will be sent again and again until it times out before CLOSE. The default value of net.ipv4.tcp_retries2 is 15, and you have to wait up to 924.6 seconds. However, this value is usually reduced.

Adjust parameters

Add the following configuration parameters to the / etc/sysctl.conf file, and finally execute sysctl-p.

Net.netfilter.nf_conntrack_icmp_timeout=10

Net.netfilter.nf_conntrack_tcp_timeout_syn_recv=5

Net.netfilter.nf_conntrack_tcp_timeout_syn_sent=5

Net.netfilter.nf_conntrack_tcp_timeout_established=600

Net.netfilter.nf_conntrack_tcp_timeout_fin_wait=10

Net.netfilter.nf_conntrack_tcp_timeout_time_wait=10

Net.netfilter.nf_conntrack_tcp_timeout_close_wait=10

Net.netfilter.nf_conntrack_tcp_timeout_last_ack=10

On how to do Linux Netfilter tuning to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.