Shulou(Shulou.com)06/01 Report--

This article mainly explains "the principle of Harbor warehouse image scanning". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn "the principle of Harbor warehouse image scanning".

The function of image scanning in the harbor repository seems high-end, but in fact, when you understand its underlying principle and process, you will find that you have done just one thing, summed up in a popular sentence, that is, to find the software packages and versions that have been installed in each image file system, and then compare it with the information released by the official system, which software version has vulnerabilities on which system version. For example, on Debian 7, what are the CVE vulnerabilities in nginx 1.12.1? by comparing the installed packages one by one, you can know how many CVE there are in the current image. Of course, if you are an unzipped version, there is nothing to say, this is similar to windows, the system can not be recognized in the control panel. The following is a brief introduction to the specific process of harbor, so that you know this function like the back of your hand.

Before we learn about image scanning, let's briefly talk about the concept of image. An image is a file system made up of many Layer tiers. The important thing is that each image has a manifest, which is similar to a concept in springboot, which means file list. An image is made up of many Layer, and this manifest file is always needed to record which layers are joined together. To scan and analyze an image, you must first get the manifest file of the image, and get the address of all the Layer images through the manifest file. Digest,digest represents an address in the docker image storage system, similar to a memory address concept in the operating system. Through this address, you can find the contents of the file. This addressable design is a major change in the v2 version. In docker hub storage system, all files have addresses, and this digest is calculated by some efficient sha algorithm.

The module in the dotted frame in the image above is the function of harbor itself. Clair is an open source system of coreos. Image scan analysis is mainly done by Clair. Its specific structure is analyzed below. Here we first focus on the analysis of harbor. The direction of the arrow roughly describes the direction of the request, and the interaction between systems may result in multiple requests.

1.UI initiates an image scan request to Job. The parameters include the warehouse name and tag.

After receiving the request, 2.Job initiates a Head request (/ v2/nginx/manifest/v1.12.1) to registry to determine whether the manifest of the current image exists, and takes out the digest of the current manifest, which is the Docker-Content-Digest stored in the response header.

3.Job inserts the digest obtained in step 2 and the warehouse name and tag into the job table as a record, and the status of job is pending.

At this time, the Job system will create a new job of the scanning task for scheduling, which involves a state machine processing flow.

The 4.Job system obtains all the mirrored Layer digest through the manifest file, encapsulates a ClairLayer parameter object for each layer, and then iteratively requests the Clair system according to the number of layers. The ClairLayer parameter structure is as follows:

Name: sha256:7d99455a045a6c89c0dbee6e1fe659eb83bd3a19e171606bc0fd10eb0e34a7dcHeaders: tokenHeader,Format: "Docker", Path: http://registry:5000/v2/nginx/blobs/7d99455a045a6c89c0dbee6e1fe659eb83bd3a19e171606bc0fd10eb0e34a7dcParentName: a55bba68cd4925f13c34562c891c8c0b5d446c7e3d65bf06a360e81b993902e1

After receiving the request, the 5.Clair system first verifies whether the parent Layer exists according to the ParentName. If it does not exist, an error is reported.

The Clair vulnerability analysis process is explained in detail below.

1. Download the image layer file

Initiate a Get request for path with the necessary headers, get an archive file, and then extract it.

two。 Probe Mirror operating system

Traverse the decompressed file directory to detect the operating system file path. First of all, you need to understand some of the basic files of each Linux distribution, such as the system version, the installed package version record, and so on.

Centos:etc/os-release,usr/lib/os-release

View the file / etc/os-release

NAME= "CentOS Linux" VERSION= "7 (Core)" ID= "centos" ID_LIKE= "rhel fedora" VERSION_ID= "7" PRETTY_NAME= "CentOS Linux 7 (Core)" ANSI_COLOR= "0 31 "CPE_NAME=" cpe:/o:centos:centos:7 "HOME_URL=" https://www.centos.org/"BUG_REPORT_URL="https://bugs.centos.org/"CENTOS_MANTISBT_PROJECT="CentOS-7"CENTOS_MANTISBT_PROJECT_VERSION="7"REDHAT_SUPPORT_PRODUCT="centos"REDHAT_SUPPORT_PRODUCT_VERSION="7"

Clair parses the file line by line, extracts the ID and VERSION_ID fields, and ends up using centos:7 as a namespace concept in clair.

two。 Probe image installed software package

Since the operating system has been detected in the previous step, you can naturally know whether the system's software management package is rpm or dpkg.

Debian, ubuntu: dpkgcentos, rhel, fedora, amzn, ol, oracle: rpm

The software management package of centos system is rpm, while the software management of debain system is dpkg.

Rpm:var/lib/rpm/Packagesdpkg:var/lib/dpkg/statusapk:lib/apk/db/installed

For example, in the debian system, you can detect which versions of the software are installed on the current system from the file / var/lib/dpkg/status file.

Package: sedEssential: yesStatus: install ok installedPriority: requiredSection: utilsInstalled-Size: 799Maintainer: Clint Adams Architecture: foreignVersion: 4.4-1Pre-Depends: libc6, libselinux1 (> = 1.32) Description: GNU stream editor for filtering/transforming text sed reads the specified files or the standard input if no files are specified, makes editing changes according to a list of commands And writes the results to the standard output.Homepage: https://www.gnu.org/software/sed/Package: libsmartcols1Status: install ok installedPriority: requiredSection: libsInstalled-Size: 257Maintainer: Debian util-linux Maintainers Architecture: amd64Multi-Arch: sameSource: util-linuxVersion: 2.29.2-1+deb9u1Depends: libc6 Description: smart column output alignment library This smart column output alignment library is used by fdisk utilities.

Parse the file line by line, extract Package and Version fields, and finally get libsmartcols1 2.29-1+deb9u1 and sed, etc.

3. Save information

Store the system version detected above and the various software package versions installed on the system in the database. The Clair system has obtained the operating system software versions of each linux version, as well as the CVE that exists in the corresponding software version. The official announcement of which CVE,Clair a software has fixed in a certain version only needs to compare the version of the software in the current image with the officially announced version. For example, if nginx 1.13.1 fixes the vulnerability CVE-2015-10203 in the officially maintained CVE information, there must be a vulnerability CVE-2015-10203 in the nginx version 1.12.1 contained in the current image. These versions are compared based on the same version of the operating system.

After Harbor's Job system sends the last layer of requests, it initiates a request query for CVE analysis results, generates an overview of the scan results and stores them in the database, mainly to record how many high-risk vulnerabilities and moderate risks have been found in the current mirror. At the same time, the status in the job table is set to finished, and if any error occurs in the request Clair, the job record is set to error. Harbor page specific vulnerability detailed data display, or through the UI system call Clair system real-time query.

Thank you for your reading. the above is the content of "the principle of Harbor Warehouse Image scanning". After the study of this article, I believe you have a deeper understanding of the principle of Harbor Warehouse Image scanning, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.