Shulou(Shulou.com)06/03 Report--

Background

With the popularity of the concept of micro services, with the continuous development of docker, we will generate a large number of docker images in the process of production, and these images will take up a lot of storage space as the version iterates. This article will analyze the factors that affect the image size, and then provide ideas for image slimming.

The relationship between Dockerfile, Docker images, and Docker containers

Inevitably, we can not avoid understanding the relationship among these three in the process of docker learning. From the perspective of R & D process, Dockerfile is the raw material of software, Docker image is the deliverable of software, and Docker container can be regarded as the running state of software. From the perspective of application software, Dockerfile, Docker image and Docker container represent three different stages of software respectively. Dockerfile is oriented to development, Docker image becomes the delivery standard, and Docker container involves deployment and operation and maintenance, all of which are indispensable and work together as the cornerstone of Docker system.

To put it simply, Dockerfile builds a Docker image and runs the Docker container through the Docker image. Later, we will have the opportunity to analyze the relationship among the three in detail.

Thin mirroring

The following is our purpose of streamlining the image:

Faster construction, smaller Docker image size, less Docker mirror layer to make full use of the mirror cache to increase Dockerfile readability to make the Docker container easier to use.

To put it simply: the most basic mirror + merge operation instructions (reduce operation instructions)

Slimming action

We are going to experiment with image packaging again and again through a series of dockerfile development in order to try our best to reduce the size of the image.

Original document

This is a very bad Dockerfile file packaged only as an analysis use case

FROM python:3.5.6-slim-stretchRUN apt-get updateRUN echo python-V RUN mkdir / code RUN mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-r requirement

Run the command: docker build-t spidermax:test_01.

The image will be packaged and docker-related programs will read the file (Dockerfile). Here is the deployment process of the mirror layer.

Sending build context to Docker daemon 289.8kB

Step 1/9: FROM python:3.5.6-slim-stretch

-- > 86669b8e5771

Step 2/9: RUN apt-get update

-- > Using cache

-- > 1b77542b82f4

Step 3ax 9: RUN echo python-V

-- > Using cache

-- > de6cc0a916d7

Step 4/9: RUN mkdir / code

-- > Using cache

-- > 996828abe04e

Step 5/9: RUN mkdir / code/db

-- > Using cache

-- > 2db7c4dfb918

Step 6/9: ADD. / code/

-- > d9d92db45b29

Step 7/9: WORKDIR / code

-- > Running in 5ae405de28fc

Removing intermediate container 5ae405de28fc

-- > a7e3696c6a5b

Step 8ax 9: RUN pip install-U pip

-- > Running in dcf5b22f6f33

Collecting pip

Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)

Installing collected packages: pip

Found existing installation: pip 18.1

Uninstalling pip-18.1:

Successfully uninstalled pip-18.1

Successfully installed pip-19.0.3

Removing intermediate container dcf5b22f6f33

-- > 949d89aabe2b

Step 9 + 9: RUN pip install-r requirement

-- > Running in 697c65e4161a

Collecting Django==2.1.7 (from-r requirement (line 1))

Downloading https://files.pythonhosted.org/packages/c7/87/fbd666c4f87591ae25b7bb374298e8629816e87193c4099d3608ef11fab9/Django-2.1.7-py3-none-any.whl (7.3MB)

Collecting EasyProcess==0.2.5 (from-r requirement (line 2))

Downloading https://files.pythonhosted.org/packages/45/3a/4eecc0c7995a13a64739bbedc0d3691fc574245b7e79cff81905aa0c2b38/EasyProcess-0.2.5.tar.gz

Collecting PyMySQL==0.9.3 (from-r requirement (line 3))

Downloading https://files.pythonhosted.org/packages/ed/39/15045ae46f2a123019aa968dfcba0396c161c20f855f11dea6796bcaae95/PyMySQL-0.9.3-py2.py3-none-any.whl (47kB)

Collecting pytz==2018.9 (from-r requirement (line 4))

Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)

Collecting selenium==3.141.0 (from-r requirement (line 5))

Downloading https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl (904kB)

Collecting urllib3==1.24.1 (from-r requirement (line 6))

Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)

Building wheels for collected packages: EasyProcess

Building wheel for EasyProcess (setup.py): started

Building wheel for EasyProcess (setup.py): finished with status' done'

Stored in directory: / root/.cache/pip/wheels/41/22/19/af15ef6264c58b625a82641ed7483ad05e258fbd8925505227

Successfully built EasyProcess

Installing collected packages: pytz, Django, EasyProcess, PyMySQL, urllib3, selenium

Successfully installed Django-2.1.7 EasyProcess-0.2.5 PyMySQL-0.9.3 pytz-2018.9 selenium-3.141.0 urllib3-1.24.1

Removing intermediate container 697c65e4161a

-- > f59692faac28

Successfully built f59692faac28

Successfully tagged spidermax:test_01

View image size through docker images

Through docker history, we will observe the packaging process of the mirror layer.

There are several missing layers, which are actually the process of automatic optimization of the docker packager program. Docker also caches each mirror layer in order to speed up mirror construction. Taking advantage of this feature can improve productivity, but it's a bit embarrassing to use it:

The previous instruction can find a mirror layer in the cache in the previous cache, and the instruction is exactly the same as the instruction to be packaged, even with spaces! If there are instructions like COPY or ADD, if the metadata of the file they refer to changes, the cache is also invalid to optimize 1

Next, we merge the same attribute instructions, such as the RUN command.

FROM python:3.5.6-slim-stretchRUN apt-get update & &\ echo python-V & &\ mkdir / code & &\ mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-U pip & &\ pip install-r requirement

The following is the compilation log

Docker build-t spidermax:test_02.

Sending build context to Docker daemon 289.8kB

Step 1/5: FROM python:3.5.6-slim-stretch

-- > 86669b8e5771

Step 2UP5: RUN apt-get update & & echo python-V & & mkdir / code & & mkdir / code/db

-- > Running in 2a879b85221f

Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]

Get:2 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [481 kB]

Ign:3 http://deb.debian.org/debian stretch InRelease

Get:4 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]

Get:5 http://deb.debian.org/debian stretch Release [118 kB]

Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [11.1 kB]

Get:7 http://deb.debian.org/debian stretch Release.gpg [2434 B]

Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7084 kB]

Fetched 7881 kB in 8s (966 kB/s)

Reading package lists...

Python-V

Removing intermediate container 2a879b85221f

-- > 5ce22d4611e9

Step 3/5: ADD. / code/

-- > af87134918ca

Step 4/5: WORKDIR / code

-- > Running in 2a5c72670be3

Removing intermediate container 2a5c72670be3

-- > 0324df92356d

Step 5ax 5: RUN pip install-U pip & & pip install-r requirement

-- > Running in 439cc1586076

Collecting pip

Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)

Installing collected packages: pip

Found existing installation: pip 18.1

Uninstalling pip-18.1:

Successfully uninstalled pip-18.1

Successfully installed pip-19.0.3

Collecting Django==2.1.7 (from-r requirement (line 1))

Downloading https://files.pythonhosted.org/packages/c7/87/fbd666c4f87591ae25b7bb374298e8629816e87193c4099d3608ef11fab9/Django-2.1.7-py3-none-any.whl (7.3MB)

Collecting EasyProcess==0.2.5 (from-r requirement (line 2))

Downloading https://files.pythonhosted.org/packages/45/3a/4eecc0c7995a13a64739bbedc0d3691fc574245b7e79cff81905aa0c2b38/EasyProcess-0.2.5.tar.gz

Collecting PyMySQL==0.9.3 (from-r requirement (line 3))

Downloading https://files.pythonhosted.org/packages/ed/39/15045ae46f2a123019aa968dfcba0396c161c20f855f11dea6796bcaae95/PyMySQL-0.9.3-py2.py3-none-any.whl (47kB)

Collecting pytz==2018.9 (from-r requirement (line 4))

Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)

Collecting selenium==3.141.0 (from-r requirement (line 5))

Downloading https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl (904kB)

Collecting urllib3==1.24.1 (from-r requirement (line 6))

Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)

Building wheels for collected packages: EasyProcess

Building wheel for EasyProcess (setup.py): started

Building wheel for EasyProcess (setup.py): finished with status' done'

Stored in directory: / root/.cache/pip/wheels/41/22/19/af15ef6264c58b625a82641ed7483ad05e258fbd8925505227

Successfully built EasyProcess

Installing collected packages: pytz, Django, EasyProcess, PyMySQL, urllib3, selenium

Successfully installed Django-2.1.7 EasyProcess-0.2.5 PyMySQL-0.9.3 pytz-2018.9 selenium-3.141.0 urllib3-1.24.1

Removing intermediate container 439cc1586076

-- > d767a8a2d9f2

Successfully built d767a8a2d9f2

Successfully tagged spidermax:test_02

Although there is no difference in size and original size.

However, in terms of image packaging, there are many fewer steps, which shortens the time of image packaging to a certain extent, which is also reflected in the packaging log just now, because the test01 version requires 9 steps, while optimizing version 1 requires only 5 steps, and the packaging efficiency is nearly doubled.

Optimization 2

The basic image we use is python:3.5.6-slim-stretch, which causes us to operate on it every time we pack. As a result, our awards will become very large. After all, the foundation has already reached 135m. Although we should not build wheels repeatedly, we can't afford such wheels.

We might as well rethink the mirroring features we use, which are nothing more than the compiler, pip and apt functions of python. As for other files such as django, which we install through requirement, and some of the dependencies that come with the compiler we don't use much (described in this way, not very rigorous), then why don't we just deploy a python environment ourselves, as long as there is apt, is there anything we can't do?

Apline image is recommended here. It's evil! It's only about 4m.

Alpine means "alpine", such as Alpine plants alpine plants, Alpine skiing alpine skiing, the alpine resort Alpine resorts.

The front page of the Alpine Linux website reads "Small! Simple! Secure! Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox." The following characteristics are summarized:

Compact: based on Musl libc and busybox, as small as busybox, the smallest Docker image is only 4MB; Security: security-oriented lightweight distribution; simple: provide APK package management tools, software search, installation, deletion, upgrade are very convenient. Suitable for container use: because of its small size and complete function, it is very suitable to be used as the basic image of the container.

The test03 version of dockerfile is as follows: FROM alpine:3.7RUN apk update & apk upgrade & &\ apk add python3 &\ python3- m ensurepip &\ if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi & &\ if [[!-e / usr/bin/python]]; then ln-sf / usr/bin/python3 / usr/bin/python; fi & &\ mkdir / code & & mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-U pip & & pip install-r requirement

Because it will be pip3 by default when installing python3, we need to change pip3 to pip to facilitate installation. In addition, we need to modify python3 to python to facilitate the startup of the image. The following is the log of the operation.

Docker build-t spidermax:test_03.

Sending build context to Docker daemon 290.3kB

Step 1/5: FROM alpine:3.7

-- > 6d1ef012b567

Step 2x5: RUN apk update & & apk upgrade & & apk add python3 & & apk add python3-dev & & python3- m ensurepip & & if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi & & if [[!-e / usr/bin/python]]; then ln-sf / usr/bin/python3 / usr/bin/python; fi & & mkdir / code & & mkdir / code/db

-- > Running in ef033831c1a9

Fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz

Fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz

V3.7.3-6-g3eacd5f9a6 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]

V3.7.3-4-g7296a289a6 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]

OK: 9049 distinct packages available

OK: 4 MiB in 13 packages

(1Action11) Installing libbz2 (1.0.6-r6)

(2can11) Installing expat (2.2.5-r0)

(3go 11) Installing libffi (3.2.1-r4)

(4 + 11) Installing gdbm (1.13-r1)

Because it will be pip3 by default when installing python3, we need to change pip3 to pip to facilitate installation. In addition, we need to modify python3 to python to facilitate the startup of the image. The following is the log of the operation.

Docker build-t spidermax:test_03.

Sending build context to Docker daemon 290.3kB

Step 1/5: FROM alpine:3.7

-- > 6d1ef012b567

Step 2x5: RUN apk update & & apk upgrade & & apk add python3 & & apk add python3-dev & & python3- m ensurepip & & if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi & & if [[!-e / usr/bin/python]]; then ln-sf / usr/bin/python3 / usr/bin/python; fi & & mkdir / code & & mkdir / code/db

-- > Running in ef033831c1a9

Fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz

Fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz

V3.7.3-6-g3eacd5f9a6 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]

V3.7.3-4-g7296a289a6 http://dl-cdn.alpinelinux.org/alpine/v3.7/community Installing xz-libs (5.2.3-r1)

(6 + 11) Installing ncurses-terminfo-base (6.0_p20171125-r1)

(7x11) Installing ncurses-terminfo (6.0_p20171125-r1)

(8x11) Installing ncurses-libs (6.0_p20171125-r1)

(9 + 11) Installing readline (7.0.003-r0)

(10 + 11) Installing sqlite-libs (3.25.3-r0)

(11 + 11) Installing python3 (3.6.5-r0)

Executing busybox-1.27.2-r11.trigger

OK: 66 MiB in 24 packages

(1x2) Installing pkgconf (1.3.10-r0)

(2Action2) Installing python3-dev (3.6.5-r0)

Executing busybox-1.27.2-r11.trigger

OK: 79 MiB in 26 packages

Requirement already satisfied: setuptools in / usr/lib/python3.6/site-packages

Requirement already satisfied: pip in / usr/lib/python3.6/site-packages

Removing intermediate container ef033831c1a9

-- > 4250724684eb

Step 3/5: ADD. / code/

-- > 8f1a939f8ac1

Step 4/5: WORKDIR / code

-- > Running in 137e7b42decb

Removing intermediate container 137e7b42decb

-- > 29586daada18

Step 5ax 5: RUN pip install-U pip & & pip install-r requirement

-- > Running in 36baec208714

Collecting pip

Downloading https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl (1.4MB)

Installing collected packages: pip

Found existing installation: pip 9.0.3

Uninstalling pip-9.0.3:

Successfully uninstalled pip-9.0.3

Successfully installed pip-19.0.3

Collecting Django==2.1.7 (from-r requirement (line 1))

Downloading https://files.pythonhosted.org/packages/c7/87/fbd666c4f87591ae25b7bb374298e8629816e87193c4099d3608ef11fab9/Django-2.1.7-py3-none-any.whl (7.3MB)

Collecting EasyProcess==0.2.5 (from-r requirement (line 2))

Downloading https://files.pythonhosted.org/packages/45/3a/4eecc0c7995a13a64739bbedc0d3691fc574245b7e79cff81905aa0c2b38/EasyProcess-0.2.5.tar.gz

Collecting PyMySQL==0.9.3 (from-r requirement (line 3))

Downloading https://files.pythonhosted.org/packages/ed/39/15045ae46f2a123019aa968dfcba0396c161c20f855f11dea6796bcaae95/PyMySQL-0.9.3-py2.py3-none-any.whl (47kB)

Collecting pytz==2018.9 (from-r requirement (line 4))

Downloading https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl (510kB)

Collecting selenium==3.141.0 (from-r requirement (line 5))

Downloading https://files.pythonhosted.org/packages/80/d6/4294f0b4bce4de0abf13e17190289f9d0613b0a44e5dd6a7f5ca98459853/selenium-3.141.0-py2.py3-none-any.whl (904kB)

Collecting urllib3==1.24.1 (from-r requirement (line 6))

Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)

Installing collected packages: pytz, Django, EasyProcess, PyMySQL, urllib3, selenium

Running setup.py install for EasyProcess: started

Running setup.py install for EasyProcess: finished with status' done'

Successfully installed Django-2.1.7 EasyProcess-0.2.5 PyMySQL-0.9.3 pytz-2018.9 selenium-3.141.0 urllib3-1.24.1

Removing intermediate container 36baec208714

-- > e59b7105697a

Successfully built e59b7105697a

Successfully tagged spidermax:test_03

If you check the size of the image, you will find that we have nearly doubled the size of the image compared to the original.

So has this achieved its goal? No, it's not enough! As an engineer, optimization never stops!

Optimization 3

Every time the image is packaged, docker will go to the cloud to request, download and install the relevant software. What will happen to the installation package after installation? -- even if it is still stored in the image, that is, the so-called cache, compared to the application layer, they will only take up our storage space and do not have too much use, so what you need is to delete the contents of this cache and simply bring it with-- no-cache, as shown in the details of apk operation.

The modified script is

FROM alpine:3.7RUN apk update--no-cache & & apk upgrade-- update-cache-- available & &\ apk add-- no-cache python3 & &\ apk add-- no-cache python3-dev & &\ python3- m ensurepip &\ if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi & &\ if [[!-e / usr/bin/python]] Then ln-sf / usr/bin/python3 / usr/bin/python; fi & &\ rm-rf / var/lib/apk/* & &\ mkdir / code & & mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-U pip & & pip install-r requirement

In this way, it will lead to a problem, that is, each package requires re-downloading resources again and again, which is actually a contradiction in efficiency, a process of playing games with each other, and changes in image size.

Of course, we can delete the cache of apk. Of course, we can delete the cache of pip. We can reduce the file size by directly deleting the installation package of pip. The code is as follows:

FROM alpine:3.7RUN apk update & & apk upgrade-- update-cache-- available & &\ apk add python3 & &\ apk add python3-dev & &\ python3- m ensurepip &\ rm-r / usr/lib/python*/ensurepip &\ if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi & &\ if [[!-e / usr/bin/python]] Then ln-sf / usr/bin/python3 / usr/bin/python; fi & &\ rm-rf / var/lib/apk/* & &\ mkdir / code & & mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-U pip & & pip install-r requirement & & rm-rf ~ / .cache/pip

The changes in the mirror are as follows:

After our optimization, we have optimized an image as high as 203m, which has been nearly doubled. In the process of compilation, we have also realized that the packing speed has been improved every time. Only in this way can we apply the idea of optimization and surpass ourselves again and again.

After modification

Finally, the information to the maintainer of the dockerfile tag said that I provided the image, and if you need help to solve any problems, you can contact me. It also means that I have the copyright to this image, although I have made it open source.

FROM alpine:3.7MAINTAINER yerikyu "yerik_shu@139.com" RUN apk update & & apk upgrade-- update-cache-- available & &\ apk add python3 &\ apk add python3-dev & &\ python3- m ensurepip &\ rm-r / usr/lib/python*/ensurepip & &\ if [!-e / usr/bin/pip]; then ln-s pip3 / usr/bin/pip; fi &\ if [!-e / usr/bin/python]] Then ln-sf / usr/bin/python3 / usr/bin/python; fi & &\ rm-rf / var/lib/apk/* & &\ mkdir / code & & mkdir / code/dbADD. / code/WORKDIR / codeRUN pip install-U pip & & pip install-r requirement & & rm-rf ~ / .cache/pip

The packaging results are as follows:

Summarize and write .dockerposts file (this is used to ignore some unwanted cache files similar to gitignore) the container only runs a single application (an embodiment of microservice) merge multiple RUN instructions into a single base image tag do not use latest each RUN instruction delete redundant files choose the appropriate base image (alpine version is best) set WORKDIR and CMD use ENTRYPOINT (optional) use exec (this entrypoint script (not reflected in this experiment) COPY and ADD first use the former to reasonably adjust the order of COPY and RUN to set default environment variables. Mapping ports and data volumes (which I set up in docker-compose, which I will discuss later) Resources

Docker Mirror layer: http://blog.daocloud.io/principle-of-docker-image/

Official information of Alpine: https://alpinelinux.org/

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.