Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to cross the pit encountered by the Docker cluster network Weave". In the daily operation, I believe that many people have doubts about how to cross the pit problem encountered by the Docker cluster network Weave. The editor consulted all kinds of materials and sorted out a simple and useful method of operation. I hope it will be helpful for everyone to answer the question of "how to cross the pit encountered by the Docker cluster network Weave". Next, please follow the editor to study!

Foreword

As one of the Docker (an open source application container engine) cross-host cluster network solution, Weave can be used to connect Docker containers deployed on multiple hosts, and applications that use the network do not have to configure port mappings, links and other information. In addition, Weave communications support encryption, and users can connect to hosts from an untrusted network.

Weave is similar to Calico in the control layer, and L2 overlay is implemented through UDP encapsulation in the data layer. Weave is implemented through usersapce before version 1.2. after the Weave-1.2 version, Weave combines the kernel Open vSwitch module to realize the function of Open vSwitch datapath (ODP). Combined with the vxlan feature of kernel, the network performance has been greatly improved.

Because ODP functions are closely integrated with kernel-related modules, some kernel-related "pits" may be encountered in practical use. Both of the issues described in this article are related to the kernel.

Pit one: network disruption caused by using Weave FastDb

Problem description

After the 1.2 version of Weave, considering the poor network performance of the original sleeve mode, the FastDb mode is added, which becomes the default mode when Weave starts. The Open vSwitch module of kernel is used in FastDb mode, and vxlan protocol is used for message encapsulation. On a CVM created with qemu-kvm, if CentOS7.0 is installed and the kernel version is kernel-3.10.123, when you start Weave and use FastDb mode, the virtio_net virtual network card will not be able to send data, resulting in network interruption of the entire virtual machine.

Analysis of problems

The reason for the network disconnection is that a kernel bug, the commit link address http://t.cn/Ro53BsH of the kernel bug, is triggered.

The bug is triggered mainly because Weave sends a 60000-byte UDP packet for PMTU detection during initialization, and the socket used by Weave is raw socket, resulting in the contamination of the memory used by virtio_net. The specific manifestation is that the host cannot be notified to vhost to obtain data, and the number of messages sent on the interface will never increase.

This problem can not only be triggered by Weave. When establishing socket with ordinary applications, raw socket is used, and the data sent is greater than the MTU value of the interface, and the UFO function of the interface is turned on. In these cases, it is very likely to trigger the problem and cause network interruption.

(figure 1:FastDb data flow principle)

Solution method

1. Upgrade the kernel to ensure that the kernel version is greater than or equal to 3.13

2. Disable the ufo feature of the virtual machine Nic.

3. CentOS7.1 's kernel-3.10.229 kernel has fixed this problem.

(figure 2:guest notifies vhost of data reading process)

Pit two: Weave cannot use FastDb mode

Problem description

On the kernel version CentOS Linux (3.10.0-327.10.1.el7.x86_64) 7 (Core), the Weave version is greater than 1.2.If the MTU value of the CVM is 1450 or less than 1474, the Fast Data Path mode cannot be selected normally when weave starts. The sleeve mode has been selected since Weave started, and the default mode should have been FastDb, which is also related to the version of the kernel.

Analysis of problems

The Fast Data Path path of Weave uses ODP technology, the OVS module in the kernel, to send packets directly to the ovs module in Container. When you start Weave, you automatically choose whether to use sleeve mode or FastDb mode, which is determined by sending a heartbeat packet. When this problem occurs, you can see the error message "FastDb timed out waiting for vxlan heartbeat" in the CVM through the Docker logs Weave log.

The heartbeat packet is a UDP packet with the destination port number of 6784. The MTU value of the interface on some CVMs is 1454, but when sending the heartbeat packet of UDP, it sends 1474 bytes, so the message will be sliced at the IP layer, and the heartbeat message will not be sent on the host. When the value of MTU is changed to 1500, it can be sent.

When the MTU is 1454, the following ICMP error message appears:

(figure 3: error ICMP message)

The ICMP message with the error above is sent by the ip_fragment function calling the ICMP_send function in the kernel:

If (unlikely (iph- > frag_off & htons (IP_DF)) & &! skb- > ignore_df) | |

(IPCB (skb)-> frag_max_size & &

IPCB (skb)-> frag_max_size > mtu)) {

IP_INC_STATS (dev_net (dev), IPSTATS_MIB_FRAGFAILS)

ICMP_send (skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED

Htonl (mtu))

Kfree_skb (skb)

Return-EMSGSIZE;}

From the above code, you can see that if an error occurs in the ICMP message, the following judgment condition iph- > frag_off & htons (IP_DF) & &! skb- > ignore_df needs to be established. Through the analysis of the crawled message, we know that the value of iph- > frag_off & htons (IP_DF) is true, then the value of skb- > ignore_ DF needs to be 0, and the key here is when the value of skb- > ignore_df is assigned to 0.

Through the analysis of the process of sending heartbeat packets by Weave, we can see that in the vxlan_tnl_send function, skb- > ignore_df is assigned to 1, and finally, when the sending function iptunnel_xmit of tunnel is called, the skb_scrub_packet function is called, and in this function, skb- > ignore_df is again assigned to 0 (kernel version: 3.10.0-327.el7). As a result, the purpose of ICMP is unreachable and the error code is an ICMP_FRAG_NEEDED message.

Void skb_scrub_packet (struct sk_buff * skb, bool xnet) {

Skb- > tstamp.tv64 = 0

Skb- > pkt_type = PACKET_HOST

Skb- > skb_iif = 0

Skb- > ignore_df = 0

Skb_dst_drop (skb)

Secpath_reset (skb)

Nf_reset (skb)

Nf_reset_trace (skb)

If (! xnet)

Return

Skb_orphan (skb)

Skb- > mark = 0

}

The above code is 3.10.0-327.el7 of CentOS7, while on some older kernel versions 3.10.0-123.el7, iptunnel_xmit calls the secpath_reset (skb) function, which does not reinitialize skb- > local_df (the lower kernel uses local_df), that is, the skb- > local_ DF value is still 1, so the above problem will not occur in this version.

Static inline void

Secpath_reset (struct sk_buff * skb) {

# ifdef CONFIG_XFRM

Secpath_put (skb- > sp)

Skb- > sp = NULL

# endif}

(figure 4: different kernel versions result in different settings)

Although this problem exists in the new kernel version, there is nothing wrong with the kernel itself. there is still a problem with the adaptation of the Weave user mode management datapath program to the kernel (it does not use ovs-switchd). In OVS, the tunnel type can be set to df_default=false for sharding.

Solution method

Ensure that the MTU value of the interface is 1500 by default.

At this point, the study on "how to cross the pit encountered by the Docker cluster network Weave" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.