Shulou(Shulou.com)06/02 Report--

In DPDK, the total number of large pages to be started is often set in the kernel startup parameters, for example, the number of large pages is set to 16, each large page is 1G, so that after the system starts, you can see the large pages allocated on node0 on / sys/devices/system/node/node0/hugepages/hugepages-1048576KB/nr_hugepages, as well as the large pages on node1. one。 Problem introduction

In DPDK, the total number of large pages to be started is often set in the kernel startup parameters, for example, the number of large pages is set to 16, each large page is 1G, so that after the system starts, you can see the large pages allocated on node0 on / sys/devices/system/node/node0/hugepages/hugepages-1048576KB/nr_hugepages, as well as the large pages on node1. By default, the kernel is evenly distributed to different socket. On my machine, there are two socket, so there will be eight large pages on each node.

However, here comes the problem. For primary processes and secondary processes using DPDK ring, in order to improve performance, it is best to bind core on the same socket. Suppose we bind multiple core on socket 0. At this time, both primary process and secondary process give priority to allocating memory on node 0 on socket 0, which leads to a problem that memory on node 1 is basically not allocated. Although a large 16-gigabyte page is set aside for DPDK applications, in fact, it is less than half of the usage rate. This is a bit of a waste of memory.

Anxious for the above problems, we expect to be able to reallocate the proportion of large pages on different socket. For example, if all DPDK processes are on socket 0, then 16g large pages can be allocated 12G to node 0, and the remaining 4G to node 1 to make full use of memory.

two。 Reconfiguration method

For different systems, the configuration method is more or less the same, for 2m large pages, can be configured directly, while for 1G large pages, the older version has some problems, such as using CentOS 6.3, dynamically modifying the number of 1G large pages is not successful.

Therefore, we only talk about the general practice here, and we can fine-tune our own system. Configure the size and number of large pages in the kernel startup parameters, taking Ubuntu as an example:

Default_hugepagesz=1G hugepagesz=1G hugepages=1 is set to GRUB_CMDLINE_ LINUX in / etc/default/grub, and then run update-grub to update the startup parameter configuration file / boot/grub/grub.cfg. After rebooting, cat / proc/meminfo can see the number of large pages and the remaining number of pages in the system (this figure is the configured 2m page).

Then there is how to solve the background problem mentioned:

After the system starts, you can adjust the number of large pages, the configuration parameter is / sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages, this is the 2m configuration, for 1G pages, nothing more than 2048K into other values. Note the node0 here, and if there are other socket, you can adjust the number of large pages on the corresponding socket.

Based on the above adjustable parameters, you can adjust the large pages on different socket after the system starts, and then start the application process of DPDK. For example, a total of 16 large pages are set, of which 12 are allocated on node 0 and 4 on node 1. There is also another question: what happens if the adjusted allocation of large pages is greater than the total number of pages? There should be no problem for 2m, but for 1G, I'm afraid it won't work. There is no experiment for now, so we need to pay attention to it. After this adjustment, all the DPDK applications are bound to the corresponding core on node 0 (because we give him more large pages), so that we can make full use of the large pages we allocate.

Original address: https://www.linuxprobe.com/dpdk-numa.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.