Shulou(Shulou.com)06/01 Report--

This article introduces you what is the principle of Calico network communication, the content is very detailed, interested friends can refer to, hope to be helpful to you.

1. Uncover the secrets of Calico network model

Below we use specific examples to help you understand the communication principle of Calico network. Randomly select a node in the k8s cluster as the experimental node, enter Container A, and check the IP address of Container A.

$ip A1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00 scope global eth0 valid_lft forever preferred_lft forever 1440 qdisc noqueue state UP link/ether 66:fb:34:db:c9:b4 brd ff:ff:ff:ff:ff:ff inet 172.17.8.2 scope global eth0 valid_lft forever preferred_lft forever

Here the container obtains the / 32-bit host address, which means that container An is used as a single point of local area network.

Take a peek at the default route for Container A:

$ip routedefault via 169.254.1.1 dev eth0169.254.1.1 dev eth0 scope link

Now the problem is, we can know from the routing table that 169.254.1.1 is the default gateway of the container, but can not find any network card corresponding to this IP address, what the heck is this?

Do not panic, first recall that when the destination address of a packet is not local, it will query the routing table. After finding the gateway from the routing table, it will first obtain the MAC address of the gateway through ARP, and then change the destination MAC to the MAC of the gateway in the network packet sent out, while the IP address of the gateway will not appear in any network packet header. In other words, no one cares what the IP address is, as long as they can find the corresponding MAC address and respond to the ARP.

With that in mind, we can move on by looking at the local ARP cache with the ip neigh command:

$ip neigh169.254.1.1 dev eth0 lladdr ee:ee:ee:ee:ee:ee REACHABLE

This MAC address should be crammed in by Calico and can respond to ARP. But how on earth did it happen?

Let's recall that under normal circumstances, the kernel will send an ARP request asking who owns the IP address 169.254.1.1 in the entire layer 2 network, and the device with this IP address will return its own MAC address to each other. But now the situation is more awkward, the container and the host do not have this IP address, and even the port calicba2f87f6bb,MAC address on the host is a useless ee:ee:ee:ee:ee:ee. According to reason, the container can not communicate with the host network at all. So how does Calico do it?

I won't beat around the bush here. In fact, Calico takes advantage of the proxy ARP function of the network card. Proxy ARP is a variant of ARP protocol. When ARP requests a target across a network segment, the gateway device receives the ARP request and returns it to the requestor with its own MAC address, which is called proxy ARP (Proxy ARP). For example:

In the picture above, the computer sends ARP to request the MAC address of the server 8.8.8.8. When the router (gateway) receives this request, it will judge that because the target 8.8.8.8 does not belong to the local network segment (that is, across the network segment), it will return its own interface MAC address to PC. When the subsequent computer accesses the server, the target MAC will be directly encapsulated as MAC254.

Now that we know that Calico essentially used the agent ARP to tell a "white lie", let's confirm it.

View the Nic information and routing information of the host:

$ip addr...771: calicba2f87f6bb@if4: mtu 1440 qdisc noqueue state UP group default link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 14 inet6 fe80::ecee:eeff:feee:eeee/64 scope link valid_lft forever preferred_lft forever...$ ip route... 172.17.8.2 dev calicba2f87f6bb scope link...

Check to see if the proxy ARP is enabled:

$cat / proc/sys/net/ipv4/conf/calicba2f87f6bb/proxy_arp1

If you are not at ease, you can verify it by grabbing tcpdump packets:

$tcpdump-I calicba2f87f6bb-e-nntcpdump: verbose output suppressed, use-v or-vv for full protocol decodelistening on calicba2f87f6bb, link-type EN10MB (Ethernet), capture size 262144 bytes14:27:13.565539 ee:ee:ee:ee:ee:ee > 0a:58:ac:1c:ce:12, ethertype IPv4 (0x0800), length 4191: 10.96.0.1.443 > 172.17.8.2.36180: Flags [P.], seq 403862039403866164, ack 2023703985, win 990, options [nop,nop,TS val 331780572 ecr 60375526] Length 412514 length 27ee:ee:ee:ee:ee:ee 13.565613 0a:58:ac:1c:ce:12 > ee:ee:ee:ee:ee:ee, ethertype IPv4 (0x0800), length 66: 172.17.8.2.36180 > 10.96.0.1.443: Flags [.], ack 4125, win 2465, options [nop,nop,TS val 603758497 ecr 331780572], length 0

Summary:

Through an ingenious method, Calico directs all the traffic of workload to a special gateway 169.254.1.1, which is diverted to the host's calixxx network equipment, and finally converts all layer 2 and layer 3 traffic into layer 3 traffic for forwarding.

The ARP reply is realized by turning on the proxy ARP function on the host, so that the ARP broadcast is suppressed on the host, the broadcast storm is suppressed, and there is no problem of ARP table expansion.

two。 Analog networking

Now that we have mastered the networking principle of Calico, we can simulate and verify it manually. The architecture is shown in the figure:

Execute the following command on Host0 first:

$ip link add veth0 type veth peer name eth0 $ip netns add ns0 $ip link set eth0 netns ns0 $ip netns exec ns0 ip an add 10.20.1.2 via 24 dev eth0 $ip netns exec ns0 ip link set eth0 up$ ip netns exec ns0 ip route add 169.254.1.1 dev eth0 scope link$ ip netns exec ns0 ip route add default via 169.254.1.1 dev eth0 $ip link set veth0 up$ ip route add 10.20.1.2 dev veth0 scope link$ ip route add 10.20.1.3 via 192.168.1.16 Dev ens192 $echo 1 > / proc/sys/net/ipv4/conf/veth0/proxy_arp

Execute the following command on Host1:

$ip link add veth0 type veth peer name eth0 $ip netns add ns1 $ip link set eth0 netns ns1 $ip netns exec ns1 ip an add 10.20.1.3 via 24 dev eth0 $ip netns exec ns1 ip link set eth0 up$ ip netns exec ns1 ip route add 169.254.1.1 dev eth0 scope link$ ip netns exec ns1 ip route add default via 169.254.1.1 dev eth0 $ip link set veth0 up$ ip route add 10.20.1.3 dev veth0 scope link$ ip route add 10.20.1.2 via 192.168.1.32 Dev ens192 $echo 1 > / proc/sys/net/ipv4/conf/veth0/proxy_arp

Network connectivity testing:

# Host0 $ip netns exec ns1 ping 10.20.1.3PING 10.20.1.3 (10.20.1.3) 56 (84) bytes of data.64 bytes from 10.20.1.3: icmp_seq=1 ttl=62 time=0.303 ms64 bytes from 10.20.1.3: icmp_seq=2 ttl=62 time=0.334 ms

The experiment was a success!

The specific forwarding process is as follows:

All packets in ns0 network space are forwarded to a virtual IP address 169.254.1.1 to send an ARP request.

When the veth side of Host0 receives the ARP request, it directly returns its MAC address to ns0 by opening the proxy ARP function of the network card.

Ns0 sends an IP packet with a destination address of ns1.

Because an address such as 169.254.1.1 is used, Host determines that it is forwarded by layer 3 route and queries the local route 10.20.1.3 via 192.168.1.16 dev ens192 is sent to the peer Host1. If BGP is configured, you will see that the proto protocol is BIRD.

When Host1 receives a packet of 10.20.1.3, it matches the local routing table 10.20.1.3 dev veth0 scope link and forwards the packet to the corresponding veth0 side, thus reaching ns1.

Similar backhaul

What is the principle of Calico network communication is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.