Shulou(Shulou.com)06/01 Report--

This article is to share with you about the use of Docker storage driver OverlayFS. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

Brief introduction

OverlayFS is a file system that is very similar to AUFS. Compared with AUFS, OverlayFS has the following features:

1) simpler design

2) since March 18th, it has entered the main line of Linux kernel.

3) it may be faster.

As a result, OverlayFS has gained rapid attention in the Docker community and is considered by many to be the successor to AUFS. As claimed, OverlayFS is still very young. Therefore, you still need to be more careful when using it in the build environment.

Docker's overlay storage driver takes advantage of many OverlayFS features to build and manage disk structures for images and containers.

Since Docker1.12, Docker has also supported overlay2 storage drivers, and overlay2 is more efficient in inode optimization than overlay. However, overlay2 drivers are only compatible with versions above Linux kernel4.0.

Note: since OverlayFS joined the kernel mainline, its name in the kernel module has been changed from overlayfs to overlay. But for the sake of distinction in this article, we use OverlayFS to represent the entire file system and overlay/overlay2 to represent the storage driver of Docker.

Image layering and sharing of overlay and overlay2OverlayFS (overlay)

OverlayFS uses two directories, places one directory on top of the other, and provides a single unified perspective. These two directories are often called layers, and this layered technique is called union mount. Technically, the lower level of the directory is called lowerdir, and the upper level is called upperdir. The unified view displayed to the outside is called merged.

The following figure shows how the Docker image and the Docker container are layered. The mirror layer is lowerdir and the container layer is upperdir. The exposed unified view is called merged.

Notice how the mirror layer and the container layer deal with the same file: the container layer (upperdir) file is explicit and hides the existence of the same file in the mirror layer (lowerdir). Container mapping (merged) shows a unified view.

The overlay driver can only work on two layers. In other words, multi-layer mirroring cannot be implemented with multi-layer OverlayFS. Instead, each mirror layer is implemented in its own directory in / var/lib/docker/overlay, using hard links, an efficient use of space, to refer to the data shared at the underlying level. Note: after Docker1.10, the directory names in the mirror layer ID and / var/lib/docker no longer correspond one to one.

Create a container, overlay drives a joint mirror layer and a new directory to the container. At the top level of the image is the read-only lowerdir in overlay, and the new directory of the container is writable upperdir.

Disk structure of mirrors and containers in overlay

The following docker pull command shows Docker host downloading a five-tier image.

$sudo docker pull ubuntuUsing default tag: latestlatest: Pulling from library/ubuntu5ba4f30e5bea: Pull complete9d7d19c9dc56: Pull completeac6ad7efd0f9: Pull completee7491a747824: Pull completea3ed95caeb02: Pull completeDigest: sha256:46fb5d001b88ad904c5c732b086b596b92cfb4a4840a3abd0e35dbb6870585e4Status: Downloaded newer image for ubuntu:latest

The output of the above figure shows that pull contains five mirror layers in five directories, each with its own directory under / var/lib/docker/overlay/. Again, as you can see, after Docker1.10, the mirror layer and the directory name no longer correspond.

$ls-l / var/lib/docker/overlay/total 20drwx-3 root root 4096 Jun 20 16:11 38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8drwx-3 root root 4096 Jun 20 16:11 55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358drwx3 root root 4096 Jun 20 16:11 824c8a961a4f5e8fe4f4243ab57c5798e7d195f688ab06ae92ba95416drwx3 root root 4096 Jun 20 ad0fe55125ebf599da124da175174a4b8c1878afe6907bf7c78570341f308461drwx- 3 root root 4096 edab9b5e5bf73f2997524eebeac1de4cf9c8b904fa8ad3ec43b3504196aa3801 16:11

In the mirror layer directory, the shared data uses hard links, and they have the same inode number. This makes effective use of the disk.

$ls-I / var/lib/docker/overlay/38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8/root/bin/ls19793696 / var/lib/docker/overlay/38f3ed2eac129654acef11c32670b534670c3a06e483fce313d72e3e0a15baa8/root/bin/ls$ ls-I / var/lib/docker/overlay/55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358/root/bin/ls19793696 / var/lib/docker/overlay/55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358/root/bin/ls

The container is also under / var/lib/docker/overlay/. Use the ls-l command to look at the container directory and find the following files and directories.

$ls-l / var/lib/docker/overlay/total 16 RW Jun 20 16:39 lower-iddrwxr-xr-x 1 root root 4096 Jun 20 16:39 mergeddrwxr-xr-x 4 root root 4096 Jun 20 16:39 upperdrwx- 3 root root 4096 Jun 20 16:39 work

These four file system objects are all components of OverlayFS. The lower-id file contains the ID at the top of the container's mirror layer.

$cat / var/lib/docker/overlay/ec444863a55a9f1ca2df72223d459c5d940a721b2288ff86a3f27be28b53be6c/lower-id55f1e14c361b90570df46371b20ce6d480c434981cbda5fd68c6ff61aa0a5358

The upper directory is the read / write layer of the container. Any changes to the container are written in this directory.

The merged directory is the mount point of the container, which is the unified view of the exposed lowerdir and the upperdir. Any changes to the container also affect this directory.

The work directory is required by the OverlayFS function and will be used by operations such as copy_up.

You can verify that the above description is correct through the mount command.

$mount | Image layering and sharing of grep overlayoverlay on / var/lib/docker/overlay/ec444863a55a.../mergedtype overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/55f1e14c361b.../root,upperdir=/var/lib/docker/overlay/ec444863a55a.../upper,workdir=/var/lib/docker/overlay/ec444863a55a.../work) OverlayFS (overlay2)

The overlay driver works on only one lower OverlayFS layer, so hard links are required for multi-layer mirroring, but the overlay2 driver natively supports multi-layer lower OverlayFS mirroring (up to 128 layers).

As a result, the overlay2 driver provides better performance in layer-related commands such as build and commit, consuming less inode than the overlay driver.

Disk structure of mirrors and containers in overlay2

Docker pull ubuntu downloads the image containing five layers, and you can see that there are six directories in / var/lib/docker/overlay2.

$ls-l / var/lib/docker/overlay2total 24drwx-5 root root 4096 Jun 20 07:36 223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7drwx-3 root root 4096 Jun 20 07:36 3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348bdrwx5 root root 4096 Jun 20 07:36 4e9fa8caffe8f8f4c8369fa7a4a9fac9573deaf481506c102d484dd1e61drwx5 root root 4096 Jun 20 88e262232ec6cb233af8293292d9530b40ff5caff3e8f8f83693fa7a4a9fac9573deaf481506c102d481506c102d484dd1e61drwx5 root root 4096 Jun 20 88e262232ec6c4af233af8293292d953f340ff5eca1e4e1694283e001f200a667bb3cb40853cf2d1b12c29feda7422fed78afeddrwx- 07:36

The l directory contains many soft connections, pointing to other layers with short names. The short name is used to avoid reaching the page size limit when the mount parameter is used.

$ls-l / var/lib/docker/overlay2/ltotal 20lrwxrwxrwx 1 root root 72 Jun 20 07:36 6Y5IM2XC7TSNIJZZFLJCS6I4I4->.. / 3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/difflrwxrwxrwx 1 root root 72 Jun 20 07:36 B3WWEFKBG3PLLV737KZFIASSW7->.. / 4e9fa83caff3e8f4cc83693fa407a4a9fac9573deaf481506c102d484dd1e6a1/difflrwxrwxrwx 1 root root 72 Jun 20 07:36 JEYMODZYFCZFYSDABYXD5MF6YO->.. / eca1e4e1694283e001f200a667bb3cb40853cf2d1b12c29feda7422fed78afed/difflrwxrwxrwx 1 root root 72 Jun 20 07:36 NFYKDW6APBCCUCTOUSYDH4DXAT->.. / 223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/difflrwxrwxrwx 1 root root 72 Jun 20 07:36 UL2MW33MSE3Q5VYIKBRN4ZAGQP->.. / e8876a226237217ec61c4baf238a32992291d059fdac95ed6303bdff3f59cff5/diff

At the lowest level, there is a link file that contains the short name for the layer mentioned earlier, and a diff directory that contains the contents of the image.

$ls / var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/diff link$ cat / var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/link6Y5IM2XC7TSNIJZZFLJCS6I4I4 $ls / var/lib/docker/overlay2/3a36935c9df35472229c57f4a27105a136f5e4dbef0f87905b2e506e494e348b/diffbin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

In the second layer, the lower file indicates the composition of the layer. The directory also has diff, merged, and work directories.

$ls / var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7diff link lower merged work$ cat / var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/lowerl/6Y5IM2XC7TSNIJZZFLJCS6I4I4 $ls / var/lib/docker/overlay2/223c2864175491657d238e2664251df13b63adb8d050924fd1bfcdb278b866f7/diff/etc sbin usr var

The directories contained in the running container also have similar files and directories. Note that in the lower file, the: symbol is used to divide the different layers, and the order is from the top to the bottom.

$ls-1 / var/lib/docker/overlay/$ cat / var/lib/docker/overlay//lowerl/DJA75GUWHWG7EWICFYX54FIOVT:l/B3WWEFKBG3PLLV737KZFIASSW7:l/JEYMODZYFCZFYSDABYXD5MF6YO:l/UL2MW33MSE3Q5VYIKBRN4ZAGQP:l/NFYKDW6APBCCUCTOUSYDH4DXAT:l/6Y5IM2XC7TSNIJZZFLJCS6I4I4

The results of mount are as follows:

$mount | grep overlayoverlay on / var/lib/docker/overlay2/9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/mergedtype overlay (rw,relatime,lowerdir=l/DJA75GUWHWG7EWICFYX54FIOVT:l/B3WWEFKBG3PLLV737KZFIASSW7:l/JEYMODZYFCZFYSDABYXD5MF6YO:l/UL2MW33MSE3Q5VYIKBRN4ZAGQP:l/NFYKDW6APBCCUCTOUSYDH4DXAT:l/6Y5IM2XC7TSNIJZZFLJCS6I4I4,upperdir=9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/diff,workdir=9186877cdf386d0a3b016149cf30c208f326dca307529e646afce5b3f83f5304/work) container uses overlay to read and write

There are three scenarios in which the container accesses the file read-only through overlay.

A file that does not exist in the container layer. If the container only reads to open a file, but the container is not in the container layer (upperdir), read it from the mirror layer (lowerdir). This will cause very little performance loss.

Files that exist only in the container layer. If the container read-only permission opens a file, and the container exists only in the container layer (upperdir) rather than the mirror layer (lowerdir), then read the file directly from the mirror layer without additional performance loss.

Files exist in both the container layer and the mirror layer. Then the file of the container layer is read because the container layer (upperdir) hides the file of the same name of the mirror layer (lowerdir). Therefore, there is no additional performance loss.

There are the following scene container modification files.

Write a file for the first time. For the first time, the container writes a file that already exists, which does not exist in the container layer. The overlay/overlay2 driver performs copy-up operations to copy files from the mirror layer to the container layer. The container then modifies the newly copied file in the container layer.

However, OverlayFS works at the file level rather than the block level. In other words, all OverlayFS copy-up operations copy the entire file, even if the file is very large, but only a small part of it is modified, which has a significant impact on container write performance. However, there are two areas worth noting:

The ▷ copy-up operation occurs only the first time the file is written. Subsequent writes to the same file are directed at the new file copied to the container layer.

▷ OverlayFS only works in two layers. This performs better than AUFS when looking in multiple layers of images.

Delete files and directories. When you delete a file, the container creates a whiteout file in the mirror layer, but the file in the mirror layer is not deleted. However, the whiteout file hides it.

Delete a directory from the container and the container layer creates an opaque directory. This is similar to a whiteout file that hides the mirror layer.

Rename the directory. Rename operations are allowed only if both the source and destination paths are at the top container level. Otherwise, EXDEV is returned.

Therefore, your application needs to be able to handle EXDEV and roll back operations, performing alternative "copy and delete" strategies.

Configure overlay/overlay2 storage driver in Docker

In order to configure the overlay storage driver for Docker, your Docker host must run on the Linux kernel3.18 version and have the overlay kernel driver loaded. For overlay2 drivers, the kernel version must be 4.0 or above. OverlayFS can run on most Linux file systems. However, using ext4 in a production environment is now the most recommended.

The following steps describe how to configure the use of OverlayFS in Docker host.

Note: before starting the configuration, if you are already using Docker daemon and have some images that you want to keep, it is easy to push them to Docker hub.

1) if Docker daemon is running, you need to stop it first.

$systemctl stop docker.service

2) check the kernel version and make sure that the kernel module of overlay is loaded.

$uname-r4.4.0-64Mutual lsmod $lsmod | grep overlayoverlay# if the above command is not output and the driver is not loaded, you can do the following $modprobe overlay

3) use the overlay/overlay2 storage driver to start Docker daemon.

Dockerd-- storage-driver=overlay2 & [1] 29403root@ip-10-0-0-174:/home/ubuntu# INFO [0000] Listening for HTTP on unix (/ var/run/docker.sock) INFO [0000] Option DefaultDriver: bridgeINFO [0000] Option DefaultNetwork: bridge

In addition, you can add the *-storage-driver=overlay flag to the DOCKER_OPTS in the Docker configuration file so that the configuration can be persisted so that you no longer need to manually specify the-storage-driver* flag when you start daemon. For example, you can persist the configuration to the configuration file / etc/default/docker and add the following to it.

DOCKER_OPTS= "--storage-driver=overlay2"

4) check whether daemon has used overlay/overlay2 storage driver.

$docker infoContainers: 0Images: 0Storage Driver: overlay2Backing Filesystem: extfs

Notice that the output shows that the back-end file system uses extfs. Although multiple file systems are supported, extfs (especially ext4) is recommended in production environments.

OverlayFS and Docker performanc

In general, the overlay/overlay2 driver is faster, almost certainly faster than aufs and devicemapper, and in some cases may be faster than btrfs. Even so, there are some things to be aware of when using the overlay/overlay2 storage driver:

Page Caching, page cache. OverlayFS supports page cache sharing, which means that if multiple containers access the same file, one or more page cache options can be shared. This makes overlay/overlay2 drivers use memory efficiently and is a good option for PaaS platforms or other high-density scenarios.

Copy_up . Like AuFS, OverlayFS needs to copy-up the first time the container modifies the file, which can cause some delay in the write operation-especially if the file to be copied is large. However, once the file has performed this up-copy operation, all subsequent operations on the file are only for the new copy of the container layer.

OverlayFS's copy_up operation is faster than AuFS's copy_up operation. Because AUFS supports more layers than OverlayFS, it can cause significant latency if you need to look up files in multiple layers.

Inode limits . Using overlay storage drivers can lead to excessive inode consumption, especially when the number of images and containers on the Docker host increases. A large number of images, or many containers, will quickly consume the inode on the Docker host. This problem does not exist with overlay2 storage drivers.

Unfortunately, you can only specify the number of inode when the file system is created. Therefore, consider placing / var/lib/docker in a separate device file system, or specifying the number of inode when creating the file system.

Here are some ways to provide the performance of OverlayFS.

Use Solid State Devices (SSD).

Use data volumes. Data volumes can improve performance because they bypass the storage driver and do not cause hidden dangers that may be caused by overconfiguration and copy-on-write.

OverlayFS compatibility

There are two points that OverlayFS is not compatible with other file systems:

Open (2). OverlayFS supports eating a subset of the POSIX standard. Take the copy-up operation as an example, your application calls fd1=open ("foo", O_RDONLY) and fd2=open ("foo", O_RDWR). Your application expects fd1 and fd2 to point to the same file, however, because of the copy-up operation, it will point to different files.

Rename (2), which is consistent with the AuFS mentioned earlier.

Thank you for reading! This is the end of this article on "what is the use of Docker storage driver OverlayFS?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it out for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.