Shulou(Shulou.com)06/03 Report--

Second, preview

Before we go into device mapper's dm dedup study, we need to preview what device mapper is and why device mapper can do block re-deletion.

1 、 device mapper

As usual, let's take a look at Wikipedia's introduction to it.

The device mapper is a framework provided by the Linux kernel for mapping physical block devices onto higher-level virtual block devices. It forms the foundation of the logical volume manager (LVM), software RAIDs and dm-crypt disk encryption, and offers additional features such as file system snapshots.

Note 1: this paragraph means that device mapper is a very important implementation of LVM and soft RAID in our linux system, as well as encryption, snapshots and other features.

Device mapper works by passing data from a virtual block device, which is provided by the device mapper itself, to another block device. Data can be also modified in transition, which is performed, for example, in the case of device mapper providing disk encryption or simulation of unreliable hardware behavior.

Note 2: this paragraph means that device mapper uses virtual devices (dm-n) to provide data transmission with other virtual or real block devices. Of course, the data can be changed (for example, raid will add redundant information), can not be changed (for example, linear is transparent), or more powerful functions, encryption, etc., we can see that this device mapper must be an evergreen tree. Anything that has something to do with storage can be implemented with it.

We have a general understanding of device mapper, and we also need to know the position of device mapper in linux I _ stack

It's the same old picture. Let's try to take a look at it and explain the other modules as well.

In view of the limited space, we will talk about it from the top to the bottom, because I have been engaged in the research of linux module O stack for 5 years, probably every position has been played, but not every module is an expert, and the knowledge stored in it is very deep, so if you want to enter the pit and store it, please think twice.

Ok, don't talk too much nonsense, let's start from the top, here is only a general analysis, not particularly deep.

I cut the picture into upper and lower halves and layered it with block layer. The above should be probably very familiar with the three blocks, LIO, virtual block devices, VFS virtual file system.

1. LIO (linux io target)

Students who do traditional SAN storage and distributed block storage should be familiar with LIO, at least they have heard of it. When we did iscsi or fc storage in the early years, we mostly used IET and SCST. Later, with the gradual improvement of LIO by linux kernel, LIO can support many kinds of fabric and backcore.

Fabric:tcm_fc (I think it may be an optical fiber card for emulex), iscsi_target_mod (docking linux kernel network protocol stack), tcm_qla2xxx (docking qlogic optical fiber card), sbp_target (IEEE 1394), tcm_usb_target (making usb mobile hard disk can be used), and so on, in which iscsi_target_mod can be given infiniband support, will become a new target for iser.

Backcore:backcore is simply the mod of virtual scsi lun, which parses and simulates almost all common standard SCSI commands to make it look the same as LUN in SCSI DISK. LIO can provide several backcore,core_file (usually a file of the file system, or directly export the block device as a file), core_iblock (can export linux block device), pscsi (can be exported directly, such as direct export / dev/sdb, etc.).

2. Vfs (virtual file system)

With a smile, I don't quite understand here, it belongs to writing blindly.

File system plays an important role in linux, and it is probably the most convenient way for users and developers to access resources.

Linux has developed a number of operable file systems for users. There are roughly the following:

Block-based fs: this type of file system usually consists of one or more block devices whose entity is a table structure that can organize data. Mapping access from file descriptors to blocks is realized by metadata, which can effectively carry out most of our daily work and life needs. We are more commonly used is ext2/3/4,xfs and btrfs and so on.

Network fs: network file system, their birth is to solve, block-based files can not effectively share files. They are all distributed in nature, because files may be used by multiple client, and this kind of file system usually needs very complex distributed protocols to support, such as NFS that we first came into contact with, smb2fs compatible with windows, and ceph, which is very popular at present. at present, ceph is not just the export protocol of NAS like its predecessors, it can provide emerging storage interface object storage and efficient block storage.

Psedudo fs: pseudo file system. Everyone should have directly performed dd and other operations on / dev/sda block devices. From the vfs point of view, / dev/sda this kind of bdev is also a pseudo file system, including our more data / proc, / sys, / configfs and other developers often use to become the use of the interface, their registration and use are very simple, can efficiently help us to complete the configuration and view of various system functions.

Special psedudo fs: this kind of special pseudo-file system is very important, and we often mount / tmp and / var as tmpfs to speed up information that does not need to be persisted to provide speed.

After roughly talking about the two LIO and vfs above, we need to talk about top block device, that is, the block decive that may be exported, which is used by LIO and VFS in linux, which is equivalent to being mount, even if the exclusive mode protects the access security of the block device.

Top block device: that is, virtual block devices, this in linux is mainly the LVM drbd md-raid bcache and device mapper that we often use, they are under / drivers/md/, they are the real connecting link between the block virtualization module, each module is quite interesting. These contents will be explained one by one in subsequent articles.

[this article was created by 51cto blogger "underlying Storage Technology" https://blog.51cto.com/12580077, official account: storage Valley], if you need to reprint, please contact me, thank you.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.