Shulou(Shulou.com)05/31 Report--

How to use DPDK to optimize VirtIO and OVS networks, for this problem, this article introduces the corresponding analysis and solutions in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Prepare the test environment

There are two nodes in total, and the configuration is basically the same. Node An is used to run virtual machines, and node B is used to test performance.

View system information

Distribution version:

$cat / etc/redhat-release CentOS Linux release 7.4.1708 (Core)

Kernel version:

$uname-aLinux osdev-gpu 3.10.0-693.17.1.el7.x86_64 # 1 SMP Thu Jan 25 20:13:58 UTC 2018 x86 "64 GNU/Linux

CPU Information:

Cat / proc/cpuinfo | tail-n26processor: 71vendor_id: GenuineIntelcpu family: 6model: 85model name: Intel (R) Xeon (R) Gold 6140 CPU @ 2.30GHzstepping: 4microcode: 0x200002ccpu MHz: 1499.941cache size: 25344 KBphysical id: 1siblings: 36core id: 27cpu cores: 18apicid: 119initial apicid: 119fpu: Yesfpu_exception: yescpuid level: 22wp: yesflags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_reqbogomips: 4604.72clflush size: 64cache_alignment: 64address sizes: 46 bits physical 48 bits virtualpower management:

Memory information:

$free-h total used free shared buff/cache availableMem: 754G 5.7G 748G 13m 425m 746GSwap: 0B 0B 0B

Nic information:

# Node Intel Corporation Ethernet Connection $lspci | grep Ethernet1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00.3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 5f:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 5f:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) $lspci | awk'$0~/Ethernet/ {printf ($1 ") Gsub (":", "\\:"); cmd= "ls / sys/bus/pci/devices/0000\\:" $1 "/ driver/module/drivers/" System (cmd)} '1a:00.0 pci:i40e1a:00.1 pci:i40e1a:00.2 pci:i40e1a:00.3 pci:i40e5f:00.0 pci:i40e5f:00.1 pci:i40e# node grep Ethernet1a:00.0 Ethernet controller $lspci | grep Ethernet1a:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00.2 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 1a:00. 3 Ethernet controller: Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) 5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 5e:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 60 for 10GbE SFP+ 00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 60 for 10GbE SFP+ 00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) $lspci | awk'$0~/Ethernet/ {printf ($1 ") Gsub (":", "\\:"); cmd= "ls / sys/bus/pci/devices/0000\\:" $1 "/ driver/module/drivers/"; system (cmd)} '1a:00.0 pci:i40e1a:00.1 pci:i40e1a:00.2 pci:i40e1a:00.3 pci:i40e5e:00.0 pci:i40e5e:00.1 pci:i40e60:00.0 pci:i40e60:00.1 pci:i40e to open IOMMU

If you want to use the VFIO driver, you must turn on IOMMU support. Enter the BIOS of the motherboard and turn on the Intel VT-d support of the processor. If you need to use single root virtualization, you need to enable SR-IOV support for PCI at the same time.

Modify the Linux kernel startup parameters and add the intel_iommu=on iommu=pt option:

$vi / etc/default/grubGRUB_CMDLINE_LINUX= "intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet"

Update the startup Grub startup configuration and restart the host:

$grub2-mkconfig-o / boot/grub2/grub.cfg$ reboot

After the host restarts, check whether the startup parameters are added correctly:

$cat / proc/cmdlineBOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=1645c2f2-2308-436e-86e6-91ebdc76477e ro intel_iommu=on iommu=pt crashkernel=auto biosdevname=0 rhgb quiet

Check whether the IOMMU in the kernel is initialized correctly, and if successful, you will have the following message:

$dmesg | grep-e DMAR-e IOMMU [0.000000] ACPI: DMAR 000000006ca320d8 002A0 (v01 ALASKA A M I 00000001 INTL 20091013) [0.000000] DMAR: IOMMU enabled [0.348112] DMAR: Host address width 46 [0.348114] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0 [0.348123] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [0.348124] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0 [0.348128 ] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [0.348129] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0 [0.348133] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [0.348134] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [0.348138] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [0.348139] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0 [ 0.348156] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df [0.348157] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0 [0.348162] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df...

Check whether the IOMMU grouping is normal. If the directory is not empty, the grouping is successful:

Ls / sys/kernel/iommu_groups/0 11 14 17 2 22 25 28 30 36 39 41 44 47 5 52 55 55 63 66 69 71 77 8 851 12 18 20 23 26 29 31 37 4 42 45 45 53 56 59 61 64 67 72 75 78 80 83 13 16 19 21 24 27 32 35 38 43 46 49 51 57 62 68 73 76 79 81 84 9 install Qemu-KVM

Add Qmu-KVM feeds:

$yum install-y epel-release$ yum install-y centos-release-qemu-ev

Install and view the Qemu-KVM version:

$yum install-y qemu-kvm-ev$ / usr/libexec/qemu-kvm-- versionQEMU emulator version 2.9.0 (qemu-kvm-ev-2.9.0-16.el7_4.14.1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers$ qemu-img-- versionqemu-img version 2.9.0 (qemu-kvm-ev-2.9.0-16.el7_4.14.1) Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers prepares virtual machine image to download Cloud image

Install dependent packages:

$yum install-y libvirt libguestfs-tools libguestfs-xfs genisoimage$ systemctl enable libvirtd & & systemctl start libvirtd & & systemctl status libvirtd

Download the Cloud image of CentOS:

$export OVS_ROOT=/opt/ovs$ mkdir-pv $OVS_ROOT/images$ wget http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2-- directory-prefix $OVS_ROOT/images

View image information:

$qemu-img info $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2image: / opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2file format: qcow2virtual size: 8.0G (8589934592 bytes) disk size: 832Mcluster_size: 65536Format specific information: compat: 0.10 refcount bits: 16$ virt-filesystems-long-parts-blkdevs-h-a $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2Name Type MBR Size Parent / dev/sda1 partition 83 8.0G / dev/sda/dev/sda device-8.0G-$virt-df-h $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 File system size% CentOS-7-x86_64-GenericCloud.qcow2:/dev/sda1 8.0G 795m 7.2G 10% extended mirror partition

Create a new mirror:

$qemu-img create-f qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2 20GFormatting'/ opt/ovs/images/CentOS-7-x86_64.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Extend the mirror partition:

$virt-resize $OVS_ROOT/images/CentOS-7-x86_64-GenericCloud.qcow2 $OVS_ROOT/images/CentOS-7-x86_64.qcow2-- expand / dev/sda1 [0.0] Examining / opt/ovs/images/CentOS-7-x86_64-GenericCloud.qcow2 25% ⟦▒▒═ ══⟧-100% ⟦▒▒▒▒⟧ --:-* Summary of changes:/dev/sda1: This partition will be resized from 8.0g to 20.0g. The filesystem xfs on / dev/sda1 will be expanded using the 'xfs_growfs' method.* [17.3] Setting up initial partition table on / opt/ovs/images/CentOS-7-x86_64.qcow2 [17.5] Copying / dev/sda1 ▒▒⟧ 00:00 Expanding / dev/sda1 using the 'xfs_growfs' methodResize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly.

View the new image information:

$qemu-img info $OVS_ROOT/images/CentOS-7-x86_64.qcow2image: / opt/ovs/images/CentOS-7-x86_64.qcow2file format: qcow2virtual size: 20g (21474836480 bytes) disk size: 834Mcluster_size: 65536Format specific information: compat: 1.1lazy refcounts: false refcount bits: 16 corrupt: false$ virt-filesystems-long-parts-blkdevs-h-a $OVS_ROOT/images/CentOS-7-x86_64.qcow2Name Type MBR Size Parent/dev/sda1 partition 83 20g / dev/sda/dev/sda device-20g-$virt-df-h $OVS_ROOT/images/CentOS-7-x86_64.qcow2 File system size Free Space% CentOS-7-x86_64.qcow2:/dev/sda1 20G 795m 19G 4% create metadata Mirror

Create a metadata configuration:

$vi $OVS_ROOT/images/meta-datainstance-id: centos7-ovs;local-hostname: centos7-ovs

Create a user data configuration (replace ssh_authorized_keys 's Key with your own):

$vi $OVS_ROOT/images/user-data#cloud-configuser: rootpassword: 123456chpasswd: {expire: False} ssh_pwauth: Truessh_authorized_keys:-ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6ijVswX40X2ercmKDZD8mVRD6GFkhdeBM/OjvGd8mSnLDdOjhZw83jhMm/rptNEDlpBW+IjxOZsDO3wm+iVcGn5LblJ3qXtdGEIHlsttFAkLsF8B3jnLBeSTRee8JXZBic5KdfYffq9JC3WrgGJl+OQz6mNW7rqquBFI98QCVlsZEvsJzw5LEm1/ej3Ka2pSkTEei+sB4PBPolnH9cUahq5T8Wwgtlw6JutNob1e5OgFWvPThTRWAtCqLaFWenedagKEA4jPseuF7dq/Eb7nEqL2jYNsWKyR1JpuUdejxAfw434guitORyvLCbj022Sgn5bwEYPIw3EVykcc6XxoZ root@osdev-gpufinal_message: "SYSTEM READY TO LOG IN"

Generate metadata mirrors:

$genisoimage-output $OVS_ROOT/images/centos7-init.iso-volid cidata-joliet-rock $OVS_ROOT/images/user-data $OVS_ROOT/images/meta-data install OVS without using DPDK's test compilation

Install dependent packages:

$yum-y install gcc autoconf automake libtool kernel kernel-devel

Get the source code:

$export OVS_ROOT=/opt/ovs & & cd $OVS_ROOT$ git clone https://github.com/openvswitch/ovs.git$ cd ovs & & git checkout-b v2.9.0/origin v2.9.0 & & git checkout-b v2.9.0/devel

Compile and install:

$. / boot.sh$ mkdir-pv $OVS_ROOT/build-nodpdk$ OVS_ROOT/target-nodpdk & & cd $OVS_ROOT/build-nodpdk$.. / ovs/configure-- enable-shared-- with-linux=/lib/modules/$ (uname-r) / build--prefix=$OVS_ROOT/target-nodpdk CFLAGS= "- g-Ofast" $make-j16 'CFLAGS=-g-Ofast-march=native' & & make install$ mkdir-pv $OVS_ROOT/target-nodpdk/modules & & cp-vf $OVS_ROOT/build-nodpdk / datapath/linux/*ko $OVS_ROOT/target-nodpdk/modules/ start and stop OVS

Initialize the environment:

$export OVS_ROOT=/opt/ovs & & export OVS_DIR=$OVS_ROOT/target-nodpdk & & export PATH=$PATH:$OVS_DIR/share/openvswitch/scripts & & cd $OVS_DIR

Start automatically using a script:

$ovs-ctl start

Use the script to stop automatically:

$ovs-ctl stop

Start manually using the command:

$mkdir-pv $OVS_DIR/var/run/openvswitch $OVS_DIR/etc/openvswitch# create ovsdb-server database: $$OVS_DIR/bin/ovsdb-tool create $OVS_DIR/etc/openvswitch/conf.db $OVS_DIR/share/openvswitch/vswitch.ovsschema# start ovsdb-server database service: $$OVS_DIR/sbin/ovsdb-server-- remote=punix:$OVS_DIR/var/run/openvswitch/db.sock-- remote=db:Open_vSwitch,Open_vSwitch Manager_options-- pidfile-- detach# initializes the database (only required for the first run): $$OVS_DIR/bin/ovs-vsctl-- no-wait init# to view dependent kernel modules: $cat / usr/lib/modules/ `uname-r` / modules.dep | awk'$1 ~ "^ extra/openvswitch" {for (iTune2) I

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.