Shulou(Shulou.com)06/02 Report--

Technical resolution series | PouchContainer supports LXCFS to achieve highly reliable container isolation

Draw the key points

Starting from this week, PouchContainer launches a series of articles on the analysis of core technical experts. The first article will provide an in-depth analysis of the business scenarios and principles applicable to LXCFS. Students who are interested in LXCFS should not miss it.

Introduction

PouchContainer is an open source container runtime product from Alibaba. The latest version is 0.3.0. The code address is https://github.com/alibaba/pouch. PouchContainer supports LXCFS from the beginning of its design to achieve highly reliable container isolation. Linux uses cgroup technology to achieve resource isolation, but the / proc file system of the host is still mounted in the container. When users read / proc/meminfo and other files in the container, they get the information of the host. The lack of / proc view isolation in the container can cause a series of problems that slow down or hinder the containerization of enterprise business. LXCFS (https://github.com/lxc/lxcfs) is an open source FUSE file system to solve the / proc view isolation problem, making the container more like a traditional virtual machine on the presentation layer. This paper first introduces the applicable business scenarios of LXCFS, analyzes LXCFS, and finally briefly introduces the work of LXCFS integration within PouchContainer.

LXCFS business scenario

In the era of physical machines and virtual machines, the company has gradually formed its own set of tool chains, such as compilation and packaging, application deployment, unified monitoring and so on. These tools have provided stable services for applications deployed in physical machines and virtual machines. Next, the role of LXCFS in the above business containerization process will be described in detail from the aspects of monitoring, operation and maintenance tools, application deployment and so on.

Monitoring and maintenance tools

Most monitoring tools rely on the / proc file system for system information. Take Alibaba as an example, part of Alibaba's basic monitoring tool is to collect information through tsar (https://github.com/alibaba/tsar)). Tsar's collection of memory and CPU information depends on the / proc file system. We can download the source code of tsar and see how tsar uses some files in the / proc directory:

1$ git remote-v

2origin https://github.com/alibaba/tsar.git (fetch)

3origin https://github.com/alibaba/tsar.git (push)

4$ grep-r cpuinfo.

5./modules/mod_cpu.c: if ((ncpufp = fopen ("/ proc/)

Cpuinfo "," r ")) = = NULL) {

6:tsar letty$ grep-r meminfo.

7./include/define.h:#define MEMINFO "/ proc/meminfo"

8./include/public.h:#define MEMINFO "/ proc/meminfo"

9./info.md: the memory counter is in / proc/meminfo, which contains some key items

10./modules/mod_proc.c: / * read total mem from

/ proc/meminfo * /

11./modules/mod_proc.c: fp = fopen ("/ proc/meminfo", "r")

12./modules/mod_swap.c: * Read swapping statistics

From / proc/vmstat & / proc/meminfo.

13./modules/mod_swap.c: / * read / proc/meminfo * /

14$ grep-r diskstats.

15./include/public.h:#define DISKSTATS "/ proc/diskstats"

The counter file for 16./info.md:IO is: / proc/diskstats, for example:

17./modules/mod_io.c:#define IO_FILE "/ proc/diskstats"

18./modules/mod_io.c:FILE * iofp

/ * / proc/diskstats*/

19./modules/mod_io.c: handle_error ("Can't open

/ proc/diskstats ",! iofp)

As you can see, tsar's monitoring of processes, IO, and CPU all depends on the / proc file system.

When the / proc file system in the container provides host resource information, this kind of monitoring cannot monitor the information in the container. In order to meet the business needs, we need to adapt to the container monitoring, and even need to develop another set of monitoring tools for the container monitoring. This change is bound to slow down or even hinder the pace of containerization of the existing business of the enterprise. Container technology should be compatible with the company's original tool chain as much as possible, taking into account the habits of engineers.

PouchContainer supports LXCFS to solve the above problems. Depending on / proc file system monitoring and operation and maintenance tools, the tools deployed in the container or on the host machine are transparent to the tools. Existing monitoring and operation and maintenance tools can be smoothly migrated to the container without adaptation or redevelopment to achieve monitoring and operation and maintenance in the container.

Next, let's take a look at the example and install PouchContainer 0.3.0 in a Ubuntu virtual machine:

Please uname-a

2Linux p4 4.13.0-36-generic

# 40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58

UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Systemd pulls pouchd. LXCFS is not enabled by default. The container created cannot use the features of LXCFS. Let's take a look at the contents of the relevant / proc files in the container:

1# systemctl start pouch

Head-n 5 / proc/meminfo

3MemTotal: 2039520 kB

4MemFree: 203028 kB

5MemAvailable: 777268 kB

6Buffers: 239960 kB

7Cached: 430972 kB

8root@p4:~# cat / proc/uptime

92594341.81 2208722.33

1 pouch run-m 50m-it registry.hub.

Docker.com/library/busybox:1.28

11 / # head-n 5 / proc/meminfo

12MemTotal: 2039520 kB

13MemFree: 189096 kB

14MemAvailable: 764116 kB

15Buffers: 240240 kB

16Cached: 433928 kB

17/ # cat / proc/uptime

182594376.56 2208749.32

As you can see, the output of / proc/meminfo and uptime files seen in the container is the same as that of the host, although the 50m memory specified when starting the container does not reflect the memory limit in the container.

Start the LXCFS service in the host, manually pull up the pouchd process, and specify the corresponding LXCFS-related parameters:

1# systemctl start lxcfs

Pouchd-D-- enable-lxcfs-- lxcfs

/ usr/bin/lxcfs > / tmp/1 2 > & 1 &

3 [1] 32707

Ps-ef | grep lxcfs

5root 698 1 0 11:08?

00:00:00 / usr/bin/lxcfs / var/lib/lxcfs/

6root 724 32144 0 11:08 pts/22

00:00:00 grep-color=auto lxcfs

7root 32707 32144 0 11:05 pts/22

00:00:00 pouchd-D-- enable-lxcfs-- lxcfs / usr/bin/lxcfs

Start the container and get the corresponding file contents:

Pouch run-- enableLxcfs-it-m 50m

Registry.hub.docker.com/library/busybox:1.28

2 / # head-n 5 / proc/meminfo

3MemTotal: 51200 kB

4MemFree: 50804 kB

5MemAvailable: 50804 kB

6Buffers: 0 kB

7Cached: 4 kB

8/ # cat / proc/uptime

910.00 10.00

Using the container started by LXCFS, read the / proc file in the container to get the relevant information in the container.

Business application

For most applications which are strongly dependent on the system, the startup program of the application needs to obtain the memory, CPU and other related information of the system, so as to configure accordingly. When the / proc file in the container can not accurately reflect the resources in the container, it will have an unnegligible impact on the above applications.

For example, for some Java applications, there is also a view / proc/meminfo in the startup script to dynamically allocate the stack size of the running program. When the container memory limit is less than the host memory, the program startup failure will occur due to the failure to allocate memory. For DPDK-related applications, the utility needs to obtain CPU information according to / proc/cpuinfo and get the CPU logic core used by the application to initialize the EAL layer. If the above information cannot be obtained accurately in the container, for DPDK applications, you need to modify the appropriate tools.

PouchContainer integrated LXCFS

PouchContainer has supported LXCFS since version 0. 1. The specific implementation can be found in: https://github.com/alibaba/pouch/pull/502.

In short, when the container starts, mount the mount point / var/lib/lxc/lxcfs/proc/ of LXCFS on the host to the virtual / proc file system directory inside the container with-v. At this point, you can see a number of proc files in the / proc directory inside the container, including meminfo, uptime, swaps, stat, diskstats, cpuinfo, and so on. The specific parameters are as follows:

1murv / var/lib/lxc/:/var/lib/lxc/:shared

2murv / var/lib/lxc/lxcfs/proc/uptime:/proc/uptime

3murv / var/lib/lxc/lxcfs/proc/swaps:/proc/swaps

4murv / var/lib/lxc/lxcfs/proc/stat:/proc/stat

5murv / var/lib/lxc/lxcfs/proc/diskstats:/proc/diskstats

6murv / var/lib/lxc/lxcfs/proc/meminfo:/proc/meminfo

7murv / var/lib/lxc/lxcfs/proc/cpuinfo:/proc/cpuinfo

To simplify use, the pouch create and run command lines provide the parameter-enableLxcfs, which can be omitted by specifying the above parameters when creating the container.

After a period of use and testing, we found that proc and cgroup will be rebuilt after lxcfs restart, resulting in a connect failed error accessing / proc in the container. In order to enhance the stability of LXCFS, in PR: https://github.com/alibaba/pouch/pull/885, the management mode of refine LXCFS is guaranteed by systemd. The specific implementation method is to do remount operation in lxcfs.service plus ExecStartPost, and traverse the container using LXCFS, and re-mount in the container.

Summary

PouchContainer supports LXCFS to achieve view isolation of in-container / proc file systems, which will greatly reduce the changes in the original tool chain and operation and maintenance habits in the process of containerization of enterprise inventory applications, and speed up the process of containerization. Strongly support the smooth transformation of enterprises from traditional virtualization to container virtualization.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.