Shulou(Shulou.com)05/31 Report--

This article introduces the knowledge of "NiLFS (2) and how to use exofs in the Linux file system". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

Using logs and objects to develop Linux file system

Linux ®continues to innovate in the field of file systems. It supports many different file systems on any operating system. It also provides cutting-edge file system technology. Linux recently introduced two new file systems, which are the NiLFS (2) journal structure file system and the exofs object-based storage system. Explore the motivations behind these two file systems and their advantages.

Develop and deploy your next application on the IBM Bluemix cloud platform.

Start the free trial now.

The announcement of a new Linux file system is always exciting and frightening. The excitement is that the file system means new space for development. Fear is because the file system was experimental in its early days and has not yet ushered in a golden age. But sometimes the announcement of a new file system also means an investment in the future of Linux, and the recent announcement of 2.6.30-rc1 does mark an interesting prospect. In the past few quarters, Linux has released three main file systems. B-Tree File System (Btrf) was introduced at the end of 2008, and recently two unique file systems have been introduced: NiLFS (2) and exofs.

Background knowledge of file system

Let's first learn about these non-traditional file systems, and then explore the details of NiLFS (2) and exofs.

Log structure file system log structure file system and SSDs

The log-structured file system is an ideal format for solid state drives (solid-state disks,SSD) made up of NAND flash memory. The basic problem with flash memory is that the number of write-erase cycles is limited. The log can be written to the entire device and fill the device as much as possible, thus minimizing the erasing cycle. For this reason, the log-structured file system performs very well on SSD (continuous writes) and provides better wear balancing.

Log structure file system has a rich history in modern computing systems. The first log-structured file system was proposed by John Ousterhout and Fred Douglis in 1988 and then implemented by the Sprite operating system in 1992. As the name implies, the log structure file system treats the file system as a circular log, writes new data and file system metadata to the head of the log, and reclaims free space from the tail (as shown in figure 1). This means that the data may appear in the log two or more times, but because the log evolves in chronological order, the most recent data is considered active data. Keeping multiple copies of the data in the log can bring some interesting benefits, which will be discussed in more detail later.

Figure 1. Schematic diagram of log structure file system

The log structure approach is not so much a selling point as an architectural detail, but it does have some unique advantages. A key advantage is data recovery after a system crash, which is easier when using the log structure method.

Another advantage is the use of the underlying storage system to mine performance. As you may recall, it is much faster to write continuously to the hard disk than to random Iripple O. If all writes are contiguous, the lookup overhead is reduced, resulting in faster hard disk Imando O and faster file system.

Object-based storage system

Traditional storage systems rely on disk drives and their local interfaces to store data persistently. These interfaces rely on block storage semantics, and small chunks of fixed-sized data are associated with their mappings (file system metadata). Object storage systems take a very different approach: instead of managing fixed-size blocks, they manage variable-sized objects and associated metadata (which provides system-level information about objects).

Object storage devices and standards

The object storage device is based on the Tmer10 Object Storage Devices (OSD) standard. The specification describes in detail the extension of the standard SCSI command set to support object-level management. In addition to defining object-level access methods, the specification also includes security and metadata management.

Object storage systems are the only way to solve scalable storage that includes multi-tenancy and security. There are many ways to build a standard OSD (see sidebar). For example, OSD-compliant components (such as OSD drives and initiators) or more advanced components (target systems that build OSD behavior on traditional drives) can be used. However, the fundamental difference between a block-based storage system and an object-based storage system is that in a block-based storage system, objects are created from a collection of blocks, which includes both data and metadata that communicates with the block using a certain protocol. In an object-based storage system, it communicates with objects and their associated metadata (see figure 2). Thus, the object storage device becomes the flat namespace (flat namespace) of the object and, if necessary, establishes a hierarchy at a higher level in the storage system stack.

Figure 2. Block-based storage system and object-based storage system

This paper explores the implementation of a file system on an object-based storage system.

Go back to the top of the page

A New implementation of Log structure File system: NiLFS (2)

NiLFS (2) is the second iteration of a log-structured file system developed by Nippon Telegraph and Telephone (NTT) in Japan. The development of this file system is very active and has recently entered the mainstream Linux kernel (as well as the NetBSD kernel). The first version of NILFS (version 1) appeared in 2005 and does not have any form of garbage collection. In 2007, version 2 was first released, including a garbage collector, and multiple snapshots can be created and maintained. This year (2009), the NiLFS (2) file system entered the mainstream kernel and can be easily enabled by installing its loadable module.

An interesting aspect of NiLFS (2) is that it supports continuous snapshot (continuous snap-shotting) technology. Because NILFS is based on the log structure, the new data is written to the head of the log, while the old data is retained (until the old data needs to be garbage collected). Because the old data is still retained, it can be rolled back on the timeline to check the different periods (epoch) of the file system. In NiLFS (2), these periods are called checkpoints, and they are an integral part of the file system. NiLFS (2) creates these checkpoints whenever changes occur, but you can also force checkpoints to be created.

File system that contains snapshots

NiLFS (2) is one of many file systems that contain snapshot behavior. Other file systems that contain snapshots are ZFS, LFS, and Ext3cow.

You can view and change checkpoints (recovery points) in snapshots. Snapshots can be mounted to the Linux file system space like other file systems, but for now they are read-only. This is useful because you can mount a snapshot and then restore a previously deleted file, or check a previous version of the file.

In addition to continuous snapshots, NiLFS (2) has many other advantages. From a usability perspective, the most important advantage is a quick restart. If the current checkpoint fails, the file system simply rolls back to the last valid checkpoint and regains the valid file system. This is obviously better than the fsck process.

Challenge of NiLFS (2) NiLFS (2) version and kernel

This NiLFS (2) demonstration is done in the 2.6.27 Linux kernel. The 2.6.30-rc1 kernel includes NiLFS (2) in mainline, but here the NILFS file system modules and tools are installed from source code. See Resources for information on how to install NiLFS (2) into the kernel.

Although continuous snapshots are a good feature, they also have some side effects. As mentioned earlier, its advantage is that it is log-structured and uses continuous writes (reducing the lookup behavior of physical disks), so it is very fast. The disadvantage is that it is log structured, so garbage collection is required to clean up old data and metadata. In general, this file system is very fast, but once garbage collection is needed, performance slows down.

Explore NiLFS (2)

Let's look at the practical application of NiLFS (2). This demo shows how to create a NiLFS (2) file system on a circular device (an easy way to test a file system), and then look at some of the features of NiLFS (2). First install the NiLFS (2) kernel module:

$sudo modprobe nilfs2 $

Next, create a file that will contain the file system (an area on the host operating system that can be mounted as a file system of the operating system itself through a circular device), and then use mkfs to build the NiLFS (2) file system (see figure 1).

Listing 1. Prepare NiLFS (2) file system $dd if=/dev/zero of=/tmp/disk.img bs=384M count=1 1: 0 records in1+0 records out402653184 bytes (403 MB) copied, 60.7253 s, 6.6 MB/s$ mkfs.nilfs2 / tmp/disk.img mkfs.nilfs2 ver 2.0Start writing file system initial data to the device Blocksize:4096 Device:/tmp/disk.img Device Size:402653184File system initialization succeeded!! $

You now have your own disk image initialized in the NiLFS (2) file system format. Next, use a loop device to mount the file system to a mount point (see listing 2). Note that when the file system is mounted, a user-space program nilfs_cleanerd is launched to provide garbage collection services.

Listing 2. Use a circular device to mount NiLFS (2) $sudo losetup / dev/loop0 / tmp/disk.img $sudo mkdir / mnt/nilfs $sudo mount-t nilfs2 / dev/loop0 / mnt/nilfs/ mount.nilfs2: WARNING!-The NILFS on-disk format may change at any time.mount.nilfs2: WARNING!-Do not place critical data on a NILFS filesystem.$ ls / mnt/nilfs $

Now, add some files to the file system, and then use the lscp command to list the currently available checkpoints (see listing 3). Use the mkcp command to define a snapshot, and then view the checkpoint again. The second time you execute the lscp command, you can see the newly created snapshot (all checkpoints and snapshots have a CNO, or checkpoint number).

Listing 3. List checkpoints and create snapshots $cd / mnt/nilfs $sudo touch file1.txt file2.txt $lscp CNO DATE TIME MODE FLG NBLKINC ICNT 1 2009-08-21 22:29:31 cp-11 3 2 2009-08-21 22:36:44 cp-11 5$ sudo mkcp-s $lscp CNO DATE TIME MODE FLG NBLKINC ICNT 1 2009-08-21 22:29:31 cp-11 3 2 2009-08-21 22:36:44 ss-11 5 3 2009-08-21 22:39:47 cp I 7 5 $

Now that you have a snapshot, use the same touch command to add some more files to the current file system (see listing 4).

Listing 4. Add more files to the NiLFS (2) file system $sudo touch file3.txt file4.txt$ ls file1.txt file2.txt file3.txt file4.txt$

Now mount the snapshot as a read-only file system. This is similar to the previous mount, but you need to specify a mounted snapshot. You can use the cp option to do this. As you can see from the previous lscp command, the snapshot is CNO=2. Use this CNO to mount the read-only file system in the mount command. After mounting, first use the ls command to list the mounted read / write file systems, and then view all the files. In a read-only snapshot, you can only see two files, which are the two files that existed when the snapshot was created (see listing 5).

Listing 5. Mount read-only NiLFS (2) snapshot $sudo mkdir / mnt/nilfs-ss2 $sudo mount.nilfs2-r / dev/loop0 / mnt/nilfs-ss2/-o cp=2$ ls / mnt/nilfs file1.txt file2.txt file3.txt file4.txt$ ls / mnt/nilfs-ss2/ file1.txt file2.txt$

Note that once checkpoints are converted to snapshots, these snapshots persist. When space needs to be cleared, the checkpoint is reclaimed from the file system, and snapshots can persist.

This demo shows two command-line utilities for NiLFS (2): lscp (listing checkpoints and snapshots) and mkcp (creating checkpoints or snapshots). There is a utility called chcp to convert checkpoints to snapshots, or to convert snapshots to checkpoints, and a rmcp utility to invalidate checkpoints or snapshots.

Given that this file system is temporary, NTT has considered some very innovative tools for the future-for example, tls (temporary ls), tdiff (temporary diff), and tgrep (temporary grep). It seems reasonable to introduce time-based functionality in the next step.

Go back to the top of the page

Extended Object File System (exofs)

Extended Object File System (exofs) is a traditional Linux file system built on object storage systems. Exofs was originally developed by IBM's Avnishay Traeger and was then called the OSD file system, or osdfs. Then Panasas, a company that builds object storage systems, took over the project and renamed it exofs (because its ancestors came from the ext2 file system).

File system on object storage system

Conceptually, object storage systems can be thought of as flat namespaces of objects and their associated metadata. In traditional block-based storage systems, metadata takes up some blocks to provide semantic glue. Figure 3 shows the architecture of exofs. Virtual File System Switch (VFS) provides a path for exofs, where exofs communicates with the object storage system through a local OSD initiator. The OSD initiator implements the OSD Tmur10 standard SCSI command set.

Figure 3. Exofs/OSD ecosystem architecture diagram

The idea behind exofs is to provide a traditional file system on OSD backing storage. This makes it easier to migrate to object-level storage because the provided file system itself is a traditional file system.

File system mapping

Each object in OSD is represented by a 64-bit identifier in a flat namespace. In order to overlay the standard POSIX interface on the object storage system, a mapping is required. Exofs provides a simple mapping that is scalable and extensible.

Files in the file system are uniquely represented by inode. Exofs maps the inode to the object identifier (OID) in the object system. In an object system, objects are used to represent all elements of the file system. Files are mapped directly to objects, and directories are also files, but files that reference files contained in the directory (in the form of file names and inode-OID pairs). Figure 4 illustrates this succinctly. There are also other objects that support inode bitmaps (for inode allocation), and so on.

Figure 4. Schematic diagram of OSD representation

The OID used to represent objects in object space is 64 bits long, so a very large number of objects are supported.

Why choose object storage?

Object storage is an interesting idea that can greatly improve the scalability of the system. It transfers parts of the file system from the host to the storage subsystem. There is a price to pay for this, but by distributing parts of the file system to multiple endpoints, you can spread the workload and make the object-based approach easier to scale to larger storage systems. The host operating system no longer needs to consider the block-to-file mapping, which is provided by the storage device itself, so the host can operate at the file level.

The object storage system also provides the ability to query available metadata. This can bring more benefits because search capabilities can be distributed to the endpoint object system.

Recently, object storage has returned in the field of cloud storage. Cloud storage providers (which sell storage as a service) provide their storage in the form of objects rather than traditional block API. These providers implement API for object transfer, management, and metadata management.

Go back to the top of the page

This is the end of "how to use NiLFS (2) and exofs in the Linux file system". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.