Shulou(Shulou.com)06/01 Report--

Network packet troubleshooting Guide-linux-like platform background information

Recently, I have been testing k8s. If you know or solve docker, you must know that most of the networks related to docker are related to bridging, routing, and Iptables. If you happen to be exposed to K8s and know how it works, you must know that kube-proxy makes iptables fly. Of course, you may think of some troubleshooting tools, such as the packet grabbing tool I used before, or the route tracking tool, but these tools are not very handy in such a complex environment, especially when packets are moving around in multiple NICs or virtual NICs on this machine, and there are many iptables strategies that allow packets to move around in kernel space. Packet grabbing tool can not catch this information, traceroute tracking routes you will find that you need to track a src,dst and port packet routing information is not achieved.

Here are some new troubleshooting tools:

IPTables trace troubleshooting local routes misarrange some network-related kernel parameter settings. Iptables tracking and troubleshooting

When it comes to Iptables debugging, I have to come up with this logical very clear diagram. I suggest that Iptables often refer to this picture when troubleshooting to see the delivery path of the packet. In my previous IPtables knowledge, I thought there was no routing operation when passing between multiple tables. As a result, the actual debugging was combined with this diagram. It turns out that Routing decision may pass between different table.

Please refer to the troubleshooting process in my article K8s Issue.

Then I have to talk about Iptables's TRACE Target. Before I learned about this Target, I used LOG Target, and found that you may not be able to track all the policies that each package passes through when you have to write a lot of IPtables, and how to deal with them.

What I'm demonstrating right now is on ubuntu:

# check whether the mod related to TRACE is loaded with modprobe nf_log_ipv4# TRACE Target can only be applied to RAW Tablesudo iptables-t raw-I PREROUTING-p tcp-m tcp-- dport 8081-j TRACEsudo iptables-t raw-I OUTPUT-p tcp-m tcp-- dport 8081-j TRACE# grep TRACE in / var/log/kern.loggrep TRACE / var/log/kern.logubuntu Ceph4:~$ grep TRACE / var/log/kern.log | grep 2213090174May 8 16:30:29 ceph4 kernel: [324781.838361] TRACE: raw:OUTPUT:policy:2 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838389] TRACE: nat:OUTPUT:rule:1 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838417] TRACE: nat:KUBE-SERVICES:rule:9 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838439] TRACE: nat:KUBE-SVC-ZP4VKUJYTBCROZYY:rule:1 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.43 .206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838454] TRACE: nat:KUBE-SEP-OR6JECCPPINGGGRC:rule:2 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.43.206.251 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838479] TRACE: filter:OUTPUT:rule:1 IN= OUT=enp3s0 SRC=192.168. 235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838493] TRACE: filter:KUBE-SERVICES:return:2 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838505] TRACE: filter:OUTPUT:rule:2 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838518] TRACE: filter:KUBE-FIREWALL:return:2 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838531] TRACE: filter:OUTPUT:rule 4 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838551] TRACE: filter:OUTPUT:policy:6 IN= OUT=enp3s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838564] TRACE: nat:POSTROUTING : rule:1 IN= OUT=enp5s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838577] TRACE: nat:KUBE-POSTROUTING:return:2 IN= OUT=enp5s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838589] TRACE : nat:POSTROUTING:rule:8 IN= OUT=enp5s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000May 8 16:30:29 ceph4 kernel: [324781.838609] TRACE: nat:POSTROUTING:policy:10 IN= OUT=enp5s0 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57266 DF PROTO=TCP SPT=18130 DPT=8081 SEQ=2213090174 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04D5CCCC0000000001030307) UID=1000 GID=1000

To explain, each trace records the table name such as raw:OUTPUT:policy:2 or nat:OUTPUT:rule:1, indicating it is Table:Chain: the explicit policy is Rule,Table and the default policy is Policy:rule number. I usually use grep ID=57266 to filter the same packet.

If you encounter routing problems in the iptables log captured above, for example, the log of my problem package is as follows, there will be no more information when the package arrives at nat:PREROUTING:policy:3, and you should have continued to enter mangle:INPUT or filter:INPUT, but there is no result. Referring to the above packet diagram, you can find that there is a route decision process here. So let me see if I rule out the problem of local routing.

Ubuntu@ceph3:~$ grep TRACE / var/log/kern.log | grep 1726587944May 8 15:51:07 ceph3 kernel: [309854.514762] TRACE: raw:PREROUTING:policy:2 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514799] TRACE: Nat:PREROUTING:rule:1 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514841] TRACE: nat:KUBE-SERVICES:rule:13 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39: Fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514861] TRACE: nat:KUBE-NODEPORTS:return:1 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514881] TRACE: nat:KUBE-SERVICES:return:14 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514897] TRACE: nat:PREROUTING:rule: 2 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514914] TRACE: nat:DOCKER:return:2 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168 .235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) May 8 15:51:07 ceph3 kernel: [309854.514930] TRACE: nat:PREROUTING:policy:3 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307) Local Route troubleshooting

In general, there are not many local routes, so the traditional method is to match the routing entries one by one, and then manually judge that they will eventually be thrown there. If no route is found to be able to handle, it will be DROP. The new version of linux uses ip rule and iproute to display and manipulate the routing table. Ip belongs to the suite in the iproute2 package. After a brief look at the document, I found that there is still a feeling of this kind of operation.

# ip rule to list ip route tablesubuntu@ceph3:~$ ip rule0: from all lookup local32766: from all lookup main32767: from all lookup default# there are three table here, first check the local numbered 0, and then check the main of 32766 Then check the default# of 32767 to list ruleubuntu@ceph3:~$ ip route list table localbroadcast 10.0.1.0 dev enp5s0 proto kernel scope link src 10.0.1.12local 10.0.1.12 dev enp5s0 proto kernel scope host src 10.0.1.12broadcast 10.0.1.255 dev enp5s0 proto kernel scope link src 10.0.1.12local 10.42.2.0 dev flannel.1 proto kernel scope host src 10.42.2.0broadcast 10 in each table. 42.2.0 dev cni0 proto kernel scope link src 10.42.2.1local 10.42.2.1 dev cni0 proto kernel scope host src 10.42.2.1broadcast 10.42.2.255 dev cni0 proto kernel scope link src 10.42.2.1broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1local 127.0.0.1 dev lo proto kernel scope host src 127 .0.0.1broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1broadcast 172.17.0.0 dev docker0 proto kernel scope link src 172.17.0.1 linkdownlocal 172.17.0.1 dev docker0 proto kernel scope host src 172.17.0.1broadcast 172.17.255.255 dev docker0 proto kernel scope link src 172.17.0.1 linkdownbroadcast 192.168.235.0 dev enp3s0 proto kernel scope link src 192.168.235.12local 192.168.235.12 Dev enp3s0 proto kernel scope host src 192.168.235.12broadcast 192.168.235.255 dev enp3s0 proto kernel scope link src 192.168.235.12 ubuntuals ceph3 dev enp3s0 onlink10.0.1.0/24 dev enp5s0 proto kernel scope link src $ip route list table maindefault via 192.168.235.2 dev enp3s0 onlink10.0.1.0/24 dev enp5s0 proto kernel scope link src 10.0.1.1210.42.0.0 dev flannel.1 onlink10 24 via 10.42.0.0 dev flannel.1 onlink10.42.1.0/24 via 10.42.1.0 42.2.0 dev cni0 proto kernel scope link src 24 via 10.42.110.42.3.0 via 10.42.3.0 dev flannel.1 onlink10.42.4.0/24 via 10.42.4.0 dev flannel.1 onlink172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown192.168.235.0/24 dev enp3s0 proto kernel scope link src 192.168.235.12ubuntualized ceph3 dev flannel.1 onlink172.17.0.0/16 dev docker0 proto kernel scope link src $ip route list table defaultubuntu@ceph3:~$

As mentioned earlier, the general method is to see whether the route entry corresponds to a specific packet one by one, but this method requires you to be very familiar with the routing rules, and it is easy to manually judge omissions. So here's a command to test routing rules, ip route get.

# lazy excerpt the instructions in man 8 ip ip route get-get a single routethis command gets a single route to a destination and prints its contents exactly as the kernel sees it.to ADDRESS (default) # the destination address.from ADDRESS # the source address.tos TOSdsfield TOS # TOS=the Type Of Service.iif NAME # the device from which this packet is expected to arrive.oif NAME # force the output device on which this packet will be routed.

Take this log in the iptables trace section above as an example

May 8 15:51:07 ceph3 kernel: [309854.514930] TRACE: nat:PREROUTING:policy:3 IN=enp5s0 OUT= MAC=00:23:7d:5b:96:ec:00:21:5a:ef:39:fe:08:00 SRC=192.168.235.13 DST=10.0.1.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28265 DF PROTO=TCP SPT=14024 DPT=8081 SEQ=1726587944 ACK=0 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A04CCCA410000000001030307)

Suppose we want to determine which route the routing of this packet will match, we can use the following command to see if the first reaction is that there is something wrong with the syntax of my command. Let's take a look at the running result after referring to rp_filter_kernel_setting at that time.

Ubuntu@ceph3:~$ ip route get from 192.168.235.13 to 10.0.1.12 iif enp5s0 tos 0x00RTNETLINK answers: Invalid cross-device link# change rp_filter kernel settingubuntu@ceph3:~$ sudo bashroot@ceph3:~# echo 2 > / proc/sys/net/ipv4/conf/default/rp_filterroot@ceph3:~# echo 2 > / proc/sys/net/ipv4/conf/all/rp_filter# Ok after changing the rp_filter kernel parameters We have a match for the same command. Root@ceph3:~# ip route get from 192.168.235.13 to 10.0.1.12 iif enp5s0 tos 0x00local 10.0.1.12 from 192.168.235.13 dev lo cache iif enp5s0root@ceph3:~#

At this point, I think you can try some other ip route get commands to see the output, such as

Ubuntu@ceph3:~$ ip route get from 172.18.0.3 to 10.0.1.12RTNETLINK answers: Invalid argument # there is no way to route from 172.18.0.3 to 10.0.1.12 ubuntualized ceph3 to 10.0.1.12RTNETLINK answers at all $ip route get from 192.168.235.3 to 10.0.1.12RTNETLINK answers: Invalid argument # cannot be routed from 192.168.235.3 to 10.0.1.12 ubuntualized ceph3 virtual machines $ip route get from 192.168.235. 12 to 10.0.1.12 # from a local card 192.168.235.12 can be routed from lo to 10.0.1.12local 10.0.1.12 from 192.168.235.12 dev lo cache ubuntu@ceph3:~$ ip route get from 192.168.235.12 to 10.0.1.13 # from a local card 192.168.235.12 can be routed from enp5s0 to 10.0.1.13 (another host Machine) 10.0.1.13 from 192.168.235.12 dev enp5s0 cache

Finally, we interpret the following routing rules in detail. For more information, please see [iproute2 doc].

Take this longer broadcast 10.0.1.0 dev enp5s0 proto kernel scope link src 10.0.1.12 as an example

The first one in broadcast 10.0.1.0 is the route type, which can be broadcast,unicast,local and so on, and if you don't write it, it is unicast,10.0.1.0 as the destination network. Dev enp5s0 stands for going out with the network card enp5s0via 10.42.3.0 you may see this sentence in some rules, which means that the next-hop gateway is 10.42.3.0proto kernel routing protocol is kernel, generated by kernel. Scope link this address is only valid on this link src 10.0.1.12 source Ip is 10.0.1.12, where 10.0.1.12 must be on the local network card address to find onlink pretending to be the next hop gateway on this link. Some network-related kernel parameter settings

OK, in the end, I have to mention the kernel parameter settings of linux, these parameters can be set in the kernel, often the essence of refinement and refinement. So here comes the question?

Q: how do I know which parameters I need?

These parameters are described in detail in the kernel documentation of A: linux, so we can read the kernel documentation, such as the parameters related to IP, to find the parameters we may need. My idea is to search the Internet through the names of kernel parameters that I think are possible, and then see what specific problems others have solved using this parameter in the results.

Q: where can I find the linux kernel documentation?

A: take ubuntu as an example. Linux-doc is the document package of the current kernel. The installed files are in the / usr/share/doc/linux-doc/ home directory. You can dpkg-L linux-doc to find the required documents. For example, zcat / usr/share/doc/linux-doc/networking/ip-sysctl.txt.gz can read the documentation on network-related kernel parameters.

Several important kernel parameters

When rp_filter is set to 2, the src of the packet is matched for all network cards. If it is matched, the packet is routed. If it is set to 1, if the network card through which the packet passes finds that the return path is not optimal, the packet is discarded.

Net.ipv4.conf.default.rp_filter = 2

Net.ipv4.conf.all.rp_filter=2

When log_martians Boolean is set to enable, when the above address is considered impossible by the kernel, the information can be recorded in the kernel log. When other supplements cross the host, the bag grabbing tool is the supplement. Knowledge needs to spread and dig deeply. You are welcome to add it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.