Shulou(Shulou.com)06/02 Report--

The knowledge about block storage, file storage and object storage has seen a good answer on Zhihu: https://www.zhihu.com/question/21536660

Easy to understand, checked some information to make a detailed supplement.

Block storage

Typical devices: disk array and hard disk block storage mainly map the whole bare disk space to the host. That is to say, for example, there are 5 hard drives in the disk array, and then N logical hard drives can be logically divided by dividing the logic disk, doing Raid, or LVM, etc. But logical disk and physical disk are two completely different concepts. Suppose that each hard disk 100g, a total of five hard disks, divided into five logical disks, each 100g, but the meaning of these five logical disks is completely different from the original five physical disks. For example, the first logical disk the first 20G may come from physical disk 1, and the second 20G may come from physical disk 2, so the logical disk is a hard disk made up of multiple physical disk logic. Block storage will then map these logical disks to the host by mapping, and the operating system on the host will recognize that there are five hard disks, but the operating system cannot tell whether it is a physical disk or a logical disk. It simply thinks that they are only five naked physical hard disks, which is no different from directly mounting a physical hard disk to the operating system, at least in the perception of the operating system. In this way, the operating system also needs to partition and format the mounted bare hard disk before it can be used, which is no different from the built-in hard disk of the ordinary host. Advantages (1) of course, the advantage of this method is that data is protected by means of Raid and LVM; (2) multiple cheap hard disks can be combined to provide services to the outside world, which is called a large-capacity logical disk, which increases the capacity; (3) when writing data, because it is a logical disk composed of multiple disks, several hard disks can be written in parallel, thus improving the efficiency of reading and writing. (4) in many cases, block storage is networked with SAN architecture, and the transmission speed and read-write efficiency are improved due to the reasons of transmission speed and encapsulation protocol. (1) when networking with SAN architecture, it is necessary to purchase additional fibre Channel cards for hosts and fiber optic switches, which is costly. (2) data cannot be shared between hosts. If the server is not clustered, the block storage bare disk is mapped to the host. After formatting, it is equivalent to the local domain for the host. Then the local disk of host A cannot be used by host B at all, and cannot share data. (3) it is not conducive to data sharing between hosts of different operating systems: because the operating system uses different file systems, after formatting, Data cannot be shared between different file systems. For example, for a win7, the file system is FAT32/NTFS, and linux is a file system that EXT4,EXT4 does not recognize NTFS.

file store

Typical devices: FTP and NFS servers have file storage in order to overcome the problem that files cannot be shared. File storage also has software and hardware integration equipment, but in fact, an ordinary PC, as long as installed with the appropriate operating system and software, you can assume that FTP and NFS services, the server after the installation of such services, is a kind of file storage. Host A can upload and download files directly to the file storage. Unlike block storage, host A no longer needs to format the file storage, because the file management function has been completed by the file storage itself. Advantages (1) low cost: any machine is fine, in addition, ordinary Ethernet is fine, there is no need for a dedicated SAN network at all, so the cost is low (2) it is convenient for file sharing (1) the read and write rate is low, the transfer rate is slow: Ethernet, the upload and download speed is slow, and all reads and writes are supported by a hard disk in a server. Compared with disk arrays, it is much slower to read and write more than a dozen or hundreds of hard drives at the same time.

Object storage

Typical equipment: the most commonly used solution for distributed server object storage with built-in large-capacity hard drives is that multiple servers have built-in large-capacity hard disks, then install object storage software, and then set up a few additional services as management nodes. Install object storage management software. The management node can manage other servers to provide read and write access. The reason for the emergence of object storage is to overcome the shortcomings of block storage and file storage and carry forward their respective advantages. To put it simply, block storage reads and writes quickly, which is not conducive to sharing, while file storage reads and writes slowly, which is conducive to sharing. Is it possible to get a read-write block to facilitate shared storage? As a result, there is object storage. First, a file contains attributes (terminology: metadata, metadata, such as the file's size, modification time, storage path, and so on) and content (data). Such as FAT32 file system, is directly a file stored together with metadata, the stored process will first file system in accordance with the smallest block size to break up (for example, 4m files, assuming that the file system requires a block 4K, then the file will be scattered as 1000 small blocks), and then write to the hard disk, there is no distinction between data and metadata. Each block will finally tell you the address of the next block to read, and then always follow the clues in this order, and finally complete the reading of all the blocks of the whole file. In this case, the reading and writing rate is very slow, because even if you have 100 robotic arms reading and writing, you can't know where the next block is until you read the first block, which means that only one robotic arm is actually working. The control node is called metadata server (server + object storage management software), which is mainly responsible for storing the attributes of objects (mainly the information that the object's data is scattered and stored in those distributed servers), while other distributed servers responsible for storing data are called OSD, which are mainly responsible for storing the data part of the file. When a user accesses an object, he will first access the metadata server. The metadata server is only responsible for feedback on where the object is stored in the OSD. Assuming that the feedback file An is stored in B, C and D OSD, then the user will directly visit the three OSD servers to read the data. At this time, because there are three OSD transmitting data at the same time, the transmission speed will be accelerated. When there are more OSD servers, the speed of reading and writing will increase. In this way, the goal of fast reading and writing can be achieved. On the other hand, object storage software has a special file system, so OSD is externally equivalent to a file server, so there is no difficulty in sharing, and it also solves the problem of file sharing, so the emergence of object storage combines the advantages of block storage and file storage.

Why does object storage have the benefits of both block storage and file storage, but also use block storage and file storage?

(1) there is a class of applications that need to store direct bare disk mapping, such as databases. Because the data needs to be mapped to itself, and then formatted according to its own database file system, it is not possible to use other storage that has been formatted into a certain file system. This type of application is more suitable for block storage. (2) the cost of object storage is higher than that of ordinary file storage, so it is necessary to purchase special object storage software and large-capacity hard disk. If the requirement for a large amount of data is not massive, but just for file sharing, it is better to use the form of file storage directly, which is cost-effective.

The following figure briefly summarizes the differences between the three:

What is the key technology of object storage file system?

(1) distributed metadata (2) concurrent data access, object storage architecture defines a new and more intelligent disk interface OSD

What is OSD?

Storage area network (SAN) and network attached storage (NAS) are two mainstream network storage architectures that we are familiar with, while object storage is a new network storage architecture. The device based on object storage technology is object storage device, abbreviated as OSD.

How do you access an object in a stored object?

In a storage device, all objects have an object identity, which is accessed through the object identity OSD command

What are the main functions of OSD?

(1) data storage. OSD manages object data and places them on a standard disk system. OSD does not provide block interface access. When Client requests data, it uses object ID and offset to read and write data. (2) Intelligent distribution. OSD optimizes data distribution with its own CPU and memory and supports data prefetching. Because OSD can intelligently support object prefetching, disk performance can be optimized. (3) the management of each object metadata. OSD manages the metadata of objects stored on it, which is similar to traditional inode metadata and usually includes the blocks of objects and the length of objects.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.