Shulou(Shulou.com)06/02 Report--

This article mainly introduces the method of dealing with the inconsistent size of file system and snapshot space. It is very detailed and has certain reference value. Friends who are interested must finish reading it.

Why is the size of the file system and snapshot space inconsistent?

When asked by users, delete files in the ECS instance and then take a snapshot, it is found that the snapshot capacity has not become smaller. The reason for this phenomenon is related to the snapshot principle and the relationship between the file system and the disk and snapshot, that is, as long as the storage block (Block) of the data has been written, even if the relevant files in the disk have been deleted, the data block will still be recorded in the snapshot, so the snapshot takes up more space than the disk. The so-called deletion in the file system is just a mark on the head of the file that needs to be deleted to let the user know that this space is available, but it does not reduce the space occupied by the disk itself.

Why is the size of the file system and snapshot space inconsistent?

What the user sees inside the instance is the disk (EBS) and what is created on the disk partition is the file system (File System). The file system is responsible for managing disk space, and its operations are eventually translated into disk Icano requests. EBS records the status of Block (the logical block address of the disk is divided into blocks of the same size by block storage) and copies dirty data to OSS as needed, which is the process of taking snapshots.

The following reasons may cause the file system space to be inconsistent with the snapshot size:

The metadata of the file system itself takes up disk space.

The process of creating a file system will write a lot of Block. In particular, slow formatting of Windows can dirty a lot of Block, while using fast formatting can reduce the number of Block writes.

In order to reduce the overhead of the file system, deleting files is only marked internally, but EBS is not aware of this deletion instruction, and Block is still allocated, so it will still put this part in the snapshot, resulting in a snapshot larger than the file system.

Modules such as Virtio-block and Xen's Block-front do not support the TRIM instruction (an IO instruction that indicates that a piece of data on the LBA is no longer in use and can be deleted), so the disk is not aware that the data can be deleted.

Principle of creating Snapshot

The principle of creating a snapshot is shown in the following figure.

Description:

The disk LBA (logical block address) is divided into Block, and the unassigned ones are called empty blocks, which are represented by a small white box in the figure.

As long as the Block has been written (allocated, it will no longer be an empty block), it will participate in the measurement.

The first snapshot of the disk is full, and the empty block is not copied.

Subsequent snapshots of the disk are incremental snapshots that copy incremental dirty data since the previous snapshot, so there may be multiple versions of the same Block in different snapshots (represented by different colors in the figure).

Snapshot measurement method

As shown in the figure above, the snapshot record space has 11 Block, 2 more than the 9 Block on disk T2.

All snapshots of a single disk are measured as a whole.

Only the allocated Block is measured. The same version of Block is counted only once at different time points.

Different versions of Block take up N times the space.

How to delete a snapshot

The principle of deleting a snapshot is shown in the following figure.

Description:

Offline analyze all Block of deleted snapshots and delete data that is not referenced by other snapshots.

After S1 is deleted, the remaining snapshots have a total of 10 Block (including 6 for S0, 2 for S1, and 2 for S2). Instead of 8 Block (6 of S0, 2 of S2).

Accurate definition of "data write operation"

First of all, to clarify a concept, for the underlying distributed storage system, "data write operation" not only means the business data written by the user, but also includes a series of Ibig O operations such as formatting disk.

After getting the disk, the user must format it before it can be used normally. The essence of formatting is the process of establishing a file system. Before a partition or disk can be used as a file system, it needs to initialize and record the data structure to the disk, a process called establishing a file system.

Therefore, establishing a file system on the disk means that there is a data write operation to the disk. Different file systems write different file sizes when formatting, as follows:

Windows system format, which is divided into fast format and normal format:

Quick formatting is simply assigning a file system to a partition and rewriting the directory table

Normal formatting not only does this, but also scans the partition sector by sector to identify and mark bad sectors.

Therefore, the normal formatting operation will fill the empty block of the disk, which is equivalent to writing the whole disk. In this case, the size of the first full snapshot is similar to the disk capacity. Fast formatting, on the other hand, takes up relatively little actual space.

Linux system formatting is also a process of establishing a file system. Therefore, for the disk that has just been formatted, when the user does not write business data, the first snapshot will also have a certain capacity, depending on the file system format.

The above is all the contents of the method of dealing with the inconsistent size of file system and snapshot space. Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.