Shulou(Shulou.com)06/02 Report--

I. Overview

The images in Docker are designed in layers, each layer can be called "layer", these layers are stored in/var/lib/docker//directory, where storage-drivers can have many types such as:AUFS, OverlayFS, VFS, Brtfs, etc. You can view the storage driver via docker info command (my system is centos 7.4):

Ubuntu systems usually default to AUFS, and centos 7.1 + uses OverlayFS. This article will introduce OverlayFS as a storage drive mirror storage principle and storage structure.

II. Introduction to OverlayFS

OverlayFS is a stacked file system that relies on and builds on other file systems (such as ext4fs and xfs, etc.), and does not directly participate in the partition of disk space structure. It only "merges" different directories in the original underlying file system and then presents them to the user. This is the joint mount technology. Compared with AUFS, OverlayFS is faster and simpler to implement. The Linux kernel provides two OverlayFS drivers for Docker: overlay and overlay2. Overlay2 is an improvement over overlay and is more efficient than overlay in terms of inode utilization. But overlay has environmental requirements: docker version 17.06.02+, host file system needs to be ext4 or xfs format.

joint mount

Overlayfs is implemented through three directories: lower directory, upper directory, and work directory. The lower directory can be multiple directories. The work directory is the work base directory. After mounting, the contents will be cleared, and the contents will not be visible to the user during use. Finally, the unified view presented to the user after joint mounting is called merged directory. The following uses mount to demonstrate how it works.

Use the mount command to mount overlayfs. The syntax is as follows:

mount -t overlay overlay -o lowerdir=lower1:lower2:lower3,upperdir=upper,workdir=work merged_dir

Create three directories A, B, C, and the worker directory:

Then mount it to/tmp/test:

Then we look at the/tmp/test directory, you will find that directories A, B, and C have been merged together, and files with the same file name will be "overwritten", where overwriting is not a real overwrite, but when two files in the directory have the same name at the time of merging, the merged layer directory will show the files closest to it:

At the same time, we can also view the mount options through the mount command:

The above method is also known as joint mount technology.

Overlay drivers in Docker

After introducing the overlay driver principle, let's look at the overlay storage driver in Docker. The following is the working principle diagram of overlay from docker official website:

In the above figure, you can see three layer structures, namely: lowerdir, upperdir, merged, where lowerdir is a read-only image layer, which is actually rootfs. Compared with the directories A and B we demonstrated above, we know that the image layer can be divided into many layers, so the corresponding lowerdir can have multiple directories. upperdir is a layer above lowerdir. This layer is the read-write layer, which is created when a container is started. All changes to the container data occur at this layer, compared to C in the example. The last merged directory is the mount point of the container, which is the unified view exposed to the user, as opposed to/tmp/test in the example. These layers are stored in/var/lib/docker/overlay2/or/var/lib/docker/overlay/(if overlay is used).

demo

Start a container

Looking at its overlay mount point, you can find its mounted merged directory, lowerdir, upperdir, and workdir:

Overlay2 can have multiple lowerdir, and it is mounted in soft connection mode, which we will explain later.

how to work

how doe that overlayfs storage driver work when data modification occur in the container? The following describes the reading and writing process:

Read:

If the file is at the container level (upperdir), read the file directly; if the file is not at the container level (upperdir), read from the mirror level (lowerdir);

Modifications:

First written: If it does not exist in upperdir, overlay and overlay2 perform copy_up operations to copy files from lowerdir to upperdir. Since overlays are file-level (copy_up behavior is generated even if the file has only a few changes), subsequent writes to the same file here will operate on copies of files that have already been copied to containers. This is often referred to as copy-on-write deletion of files and directories: when files are deleted in containers, whiteout files are created at the upper dir level, files at the lower dir level are not deleted because they are read-only, but without files prevents them from being displayed, and when directories are deleted in containers, an opaque directory at the upper dir level, which, like whiteout above, prevents users from continuing to access them, even if the mirror level is still present.

precautions

copy_up operation only occurs when the file is first written, and only copies are modified thereafter,overlayfs only applies to two levels of directories, and search is faster than AUFS. The deletion of files in the container layer is only a "blind method", which is blocked by whiteout files. The image layer is not deleted, which is why the images submitted and saved by docker commit will become larger and larger. No matter how the data is deleted in the container layer, the image layer will not change.

3. overlay2 mirror storage structure

Pulling an ubuntu image from the repository, the result shows that a total of 4 layers of images have been pulled as follows:

The 4 layers are stored in/var/lib/docker/overlay2/:

There is an l directory containing all layers of soft links, short links use short names to avoid parameters reaching page size limits when mounting (short directories when viewing mount commands in the demo):

The mirror directory at the bottom contains a diff directory that stores the mirror contents of the current layer and a link file with the corresponding short name:

Above this mirror, there are also work directories and lower files. The lower file is used to record the short name of the parent layer, and the work directory is used to jointly mount the specified work directory. And how are these directory and mirror relationships organized together? The answer is through metadata associations. Metadata is divided into image metadata and layer metadata.

image metadata

The image metadata is stored in/var/lib/docker/image//imagedb/content/sha256/directory. The name is a file named after the image ID. The image ID can be viewed through docker images. These files store the rootfs information of the image in the form of json, the image creation time, the construction history information, the container used, including the startup Entrypoint and CMD, etc. For example, the id of ubuntu image is 47b19964fb50:

View its corresponding metadata (using vim :%! python -m json.tool formatted as json) intercepts its rootfs components:

The diff_id above corresponds to a mirror layer, which is arranged in order, from top to bottom, indicating the lowest layer to the top layer of the mirror layer:

diff_id How do I associate layers? Specifically, docker uses each diff_id and history information in rootfs to calculate the corresponding content addressing index (chainID), and chaiID is associated with the layer layer, which in turn is associated with the mirror file of each mirror layer.

layer metadata

Layer corresponds to the concept of mirror layer. Before docker version 1.10, mirrors were managed through a graph structure. Each mirror layer had metadata, recording the construction information of the layer and the ID of the parent mirror layer, while the topmost mirror layer recorded some more information as the metadata of the entire mirror. graph maintains a tree-like mirror layer structure based on the mirror ID(i.e., the mirror layer ID of the top layer) and the mirror layer ID of the parent record of each mirror layer.

One of the big changes to image metadata management since docker 1.10 is the simplification of image layer metadata, which consists of only one specific image layer package. After a user downloads an image layer on the docker host, docker builds local layer metadata on the host based on the image layer package and image metadata, including diff, parent, size, etc. When docker uploads a new mirror layer generated on the host to registry, the metadata associated with the new mirror layer on the host is not packaged with the mirror layer.

Docker defines two interfaces, Layer and RWLayer, which are used to define some operations of read-only layer and read-write layer respectively, and roLayer and mountedLayer are defined to implement the above two interfaces respectively. roLayer is used to describe immutable mirror layers and mountedLayer is used to describe read-write container layers. Specifically, the contents stored in roLayer mainly include chainID, diffID, parent, cacheID and size of the mirror layer. This metadata is stored in the/var/lib/docker/image//layerdb/sha256//folder. As follows:

There are three files in each chainID directory cache-id, diff, and zize:

cache-id file:

docker randomly generated uuid, the content is the directory index to save the mirror layer, that is, the directory in/var/lib/docker/overlay2/, which is why the corresponding layer directory can be found through chainID. ChainID d801a12f6af7beff367268f99607376584d8b2da656dcd8656973b7ad9779ab4 corresponds to directory 130ea10d6f0ebfafc8ca260992c8d0bef63a1b5ca3a7d51a5cd1b1031d23efd5,/var/lib/docker/overlay2/130ea10d6f0ebfafc8ca260992c8d0bef63a1b5ca3a7d51a5cd1b1031d23efd5

diff file:

diff_id in the mirror metadata is saved (corresponds to uuid in diff_ids in metadata)

size file:

Saved the size of the mirror layer

Among all the attributes of layer, diffID is calculated using SHA256 algorithm based on the contents of the image layer file package. ChainID is an index based on the content store, which is calculated from the diffID of the current layer and all ancestor mirror layers, as follows:

If the mirror layer is the lowest (no parent mirror layer), the diffID of that layer is the chainID. The formula for calculating the chainID of the mirror layer is chainID(n)= SHA256 (chain(n-1) diffID(n)), that is, the SHA256 check code is calculated according to the chainID of the parent mirror layer plus a space and the diffID of the current layer.

MountedLayer information stores readable init layer and container mount point information including: container init layer ID (init-id), ID used for joint mount (mount-id), and chainID(parent) of parent layer image of container layer. The files are located in/var/lib/docker/image//layerdb/mounts//. Start a container with id 3c96960b3127 as follows:

Look at the three files that correspond to mountedLayer:

You can see that the initID is the name of the directory stored in/var/lib/docker/overlay2/, followed by-init after mountID:

You can also view the mountID directly through the mount command, corresponding to the/var/lib/docker/overlay2/directory, which is also the merged directory presented by overlayfs:

A file is created in the container:

At this point, you can see the corresponding file in the merged directory of the host:

About the init layer

The init layer ends with a uuid+-init, sandwiched between the read-only layer and the read-write layer, and is used to store information such as/etc/hosts and/etc/resolv.conf. The reason why this layer is needed is that when the container is started, these files or directories that should belong to the image layer, such as hostname, need to be modified by the user, but the image layer is not allowed to be modified. Therefore, when starting, mount a separate init layer, and modify the files in the init layer to modify these files. These changes tend to read only the current container and do not commit the init layer when the docker commit is a mirror. /var/lib/docker/overlay2//diff

summary

Through the above content introduction, a complete layer of a container should be composed of three parts, as shown in the following figure:

Image layer: also known as rootfs, provides the file system for container startup. Init layer: used to modify some files in the container, such as/etc/hostname,/etc/resolv.conf, etc. Container layer: unified read-write directory provided to users by using joint mount.

IV. Summary

This paper introduces the principle of mirror storage with overlay fs as storage driver. The mirror data of each layer is stored in/var/lib/docker/overlay2//diff directory, the init layer data is stored in/var/lib/docker/overlay2//diff directory, and the final unified view (container layer) data is stored in/var/lib/docker/overlay2//diff directory. Docker organizes these directories by using content addressing (chainID) through image metadata and layer metadata to form the file system on which the container runs.

Reference:

《use overlayfs driver 》

Docker Image Storage Management

This article on the use of Docker image storage overlayfs introduced here, more related Docker image storage overlayfs content please search the previous article or the following related articles, I hope you will support more in the future!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.