Shulou(Shulou.com)06/02 Report--

I. Background

As mentioned in the previous article, in the centralized network node mode, all computing nodes only install Layer 2 agents, and all Layer 3 traffic must pass through the network nodes regardless of whether it is north-south or east-west. Although HA can ensure high availability of network nodes, only one network node is working at the same time in the HA mode based on vrrp. In this way, network nodes will still become performance bottlenecks in large-scale scenarios. For this reason, the DVR mode launched by the openstack community from the Juno version to solve the above problems needs to be explained: DVR and L3 HA functions cannot be enabled at the same time before the Mitaka version, and DVR and L3 HA functions are supported to be enabled at the same time after the Mitaka version.

II. DVR Introduction

In order to solve the traffic bottleneck problem of network nodes, DVR deploys L3 agents on compute nodes, so that east-west traffic between different subnets and north-south traffic bound to vm with floating ip directly access the external network through compute nodes. Only north-south traffic bound to vm without floating ip needs to access the external network through network nodes SNAT. At this time, the cluster architecture is shown as follows:

Unlike centralized network nodes where all compute nodes only go through Layer 2 traffic, DVR mode allows each compute node to go through Layer 3 traffic to apportion the traffic pressure on network nodes.

III. Internal components of the network and compute nodes 1. Network nodes

In DVR mode, the internal components of the network node are shown in the following figure:

As you can see, the network node with DVR mode enabled has an additional SNAT Namespace space. When DVR function is enabled on all compute nodes, Metadata Agent in Router Namespace is only responsible for processing metadata in Project network, SNAT Namespace is responsible for accessing external network through source address translation for vm with only fix ip. If DVR mode is turned off for all compute nodes, vm traffic is consistent with centralized network nodes, that is, all Layer 3 traffic needs to be processed through Router Namespace of network nodes.

2. computing node

When DVR is enabled, the internal components of the compute node are shown in the following figure:

DVR-enabled compute nodes have Distributed Router NameSpace because they are deployed with L3 Agent, and fip namespace is automatically generated when vm is created. Distributed Router NameSpace of all compute nodes is completely consistent, and ip and mac addresses of namespace interfaces are also the same.(The namespaces of all compute nodes are copies of network nodes during initialization). Everyone who knows the network knows that ip and mac addresses in the same network at the same time should be consistent, otherwise there will be conflicts in arp table entries learned by exchanging mac addresses repeatedly. To solve this problem, DVR structure specifies a globally unique mac address (dvr_host_mac) for each compute node running L3 Agent.

IV. East-west traffic analysis of vm

The traffic trend between VMs under the same subnet is similar to that of centralized network nodes, and will not be described here. The following is an example of the vxlan traffic trend between VMs under different subnets. At this time, the traffic trend between VMs is shown in the following figure:

1. vm1 in compute1 makes a request to vm2 in compute2. In this case, the source/destination ip is the ip address of vm1/2, and the source/destination mac address is the mac address of vm1 and gateway qr-1.

2. Messages are checked for iptables security by linux bridge and sent to br-int.

3. Messages entering br-int are marked with an internal vlan number and sent to vm1 gateway qr-1. The gateway address of vm1 is configured on qr-1 interface, and the message flows out from qr-2 interface through table lookup. The gateway address of vm2 is set on qr-2 interface.

4. For the message coming out of qr-2 port, the source/destination ip is ip address of vm2 gateway (qr-2) and ip address of vm2, the source/destination mac is mac address of qr-2 port and mac address of vm2, and the message enters br-tun.

5. The message changes the source mac address (qr-2) to the globally unique mac address (dvr_host_mac) on the br-tun switch, and then performs vxlan encapsulation to leave compute1.

6. After the message arrives at Compute2, the VXLAN is first decapsulated, and then the source MAC address (dvr_host_mac) is changed to the VM2 gateway (qr-2) MAC address, sent to the br-int and marked with the internal VLAN number in the br-int switch.

7. Remove the internal vlan number of the message and enter the linux bridge to check the security policy.

8. The final data message reaches VM2.

The return process of the VM2 data message is consistent with the process of the data message arriving at VM2, and will not be described again.

V. North-South flow analysis of vm

vm north-south traffic is divided into floating ip and fix ip, and these two cases are explained respectively:

1. fix ip access to external network

VM without floating ip binding needs to perform address translation through SNAT Router NameSpace of network node when accessing external network, and its traffic trend is shown in the following figure:

1. vm initiates a request to the external network, and the data message is sent to the linux bridge.

2. The data message entering linux bridge is sent to br-int after iptables security policy check, and the internal vlan is marked at this time.

3. The data packet is sent from br-int to qr port of Router NameSpace. The gateway address of vm is configured on this interface. The mac address of sg port of Snet NameSpace is parsed in Router NameSpace. The sg interface is the interface of the subnet where vm is located. The ip address on this interface is on the same network segment as vm. The message is then sent to br-tun.

4. After the data message enters br-tun, the internal vlan number is removed, the vxlan encapsulation is carried out, the vni number is marked, and the data message leaves conpute1.

5. The data packet enters the Network node, removes the vni number, performs vxlan decapsulation, and is sent to the br-int switch. After entering the br-int switch, the internal vlan number is marked.

6. After the data packet enters SG, it performs routing lookup table and sends the data to FG port. The configuration on FG port is routable public network IP.

7. SNAT address conversion is performed on the fg port of the data packet, and the converted source IP address is the public IP configured on the fg port to access the public network.

2. floati ip access to external network

After DVR function is enabled, each compute node host is installed with L3 Agent. VMs bound with floating ip no longer need to detour to network nodes. The compute node host directly accesses the public network. The traffic trend is as shown in the following figure:

1. vm initiates access to the external network. since vm is a private network address of provider type, we must first find the gateway where vm address is located.

2. After passing through linux bridge and br-int, the data packet enters qr port of Distribute NameSpace, and the ip address configured on this interface is the gateway address of vm.

3. The data packet flows out of the qr port and enters the rfp port. Two ip addresses are configured on this interface, where 3 is the floating ip address bound to vm. SNAT address translation is performed here. When external network traffic accesses vm, DNAT address translation is performed in this namespace by using iptables.

4. Through the direct connection interface (4) of qrouter and fip internal communication, the interface address is maintained by L3 Agent itself, ip is in the format of 169.254.x.x/31, and the data packet is sent to fip namespace.

5. After receiving the data packet, the directly connected interface fpr of the fip space forwards it to the fg port of the external network gateway.

6. The fip namespace external network gateway interface sends the data packet to the br-ex switch and finally accesses the internet through the physical NIC. The data flow direction of the external network access vm is the reverse direction of this process, which is not described here.

3. precautions

For floating ip packets in and out of the need to pay attention to the following places:

1.fg interface will be configured with an additional IP address of the external network, which is why VM IP address of the external network will not be directly set to the IP address of the public network in the public cloud scenario, because each computing host needs an additional address as fg gateway address.

2. When an external network accesses vm, the IP address requested is the IP address doing SNAT on the rfp interface in the qrouter namespace. However, at this time, the fg interface will respond to the arp address resolution request of the external IP on the rfp interface, so it is generally considered that the fg interface is the arp proxy interface of the floating IP.

VI. Network Node HA

From the foregoing, it is known that network nodes in DVR mode only perform SNAT address translation for vm without floating ip binding, and qrouter namespace only processes metadata, so unlike traditional L3HA high availability for Router NameSpace, L3HA under DVR is high availability for SNAT NameSpace, which is still implemented by vrrp, as shown in the following figure:

From the perspective of deployment structure, the IP address of SNAT external network and IP address of subnet interface should be made highly available respectively, so when keepalive is used, the architecture is as shown in the following figure:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.