Shulou(Shulou.com)06/01 Report--

This article is about what ZFS refers to in linux. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.

History of ZFS

The Z file system (Z File System) (ZFS) was developed by Matthew Ahrens and Jeff Bonwick in 2001. ZFS is designed as the next generation file system of OpenSolaris of Sun Microsystems (Sun MicroSystem). In 2008, ZFS was ported to FreeBSD. In the same year, a project to migrate ZFS to Linux was also launched. However, because ZFS is licensed under the General Development and Distribution license (Common Development and Distribution License) (CDDL), it is not compatible with the GNU General Public license, so it cannot be migrated to the Linux kernel. To solve this problem, most Linux distributions provide some ways to install ZFS.

Shortly after Oracle acquired Sun Microsystems, OpenSolaris was closed, which made the subsequent development of ZFS closed. Many ZFS developers are very unhappy with this. 2/3 of ZFS core developers, including Ahrens and Bonwick, left Oracle because of this decision. They joined other companies and founded the OpenZFS program in September 2013. The project leads the open source development of ZFS.

Let's go back to the license issue mentioned above. Now that the OpenZFS project is separate from Oracle, one might wonder why they don't use a license that is compatible with GPL, so they can add it to the Linux kernel. According to the OpenZFS website, changing the license requires contacting and obtaining permission from all those who have contributed code to the current OpenZFS implementation (including the initial public ZFS code and OpenSolaris code). This is almost impossible (because some contributors may have died or are hard to find), so they decided to keep the original license.

What is ZFS and what are its features?

As mentioned earlier, ZFS is an advanced file system. Therefore, it has some interesting features. For example:

Storage pool

Copy while writing

Snapshot

Data integrity verification and automatic repair

RAID-Z

The maximum single file size is 16 EB (1 EB = 1024 PB)

Storage of up to 256 terabytes (256 to 1015) ZB (1 ZB = 1024 EB)

Let's take a closer look at some of these features.

How do I install ZFS?

If you want to use ZFS immediately (out of the box), you need to install FreeBSD or an operating system that uses the illumos kernel. Illumos is a cloned version of the OpenSolaris kernel.

In fact, support for ZFS is the main reason why some experienced Linux users choose BSD.

If you want to try ZFS on Linux, you can only use it on storage file systems. As far as I know, no Linux distribution can install ZFS in the root directory and use it right out of the box. If you are interested in trying ZFS on Linux, there are plenty of tutorials on the ZFS on Linux project that will teach you how to do it.

Storage pool

Unlike most file systems, ZFS combines the features of a file system and a volume manager. This means that unlike other file systems, ZFS can create file systems that span a series of hard drives or pools. Not only that, you can also increase the storage capacity of the pool by adding hard drives. ZFS can be partitioned and formatted

Ten reasons and characteristics of using ZFS

1. No more fsck, scandisk.

Whether you are using Linux,UNIX or Windows, I believe we all have a similar experience: when the system is accidentally powered off or shut down illegally, the file system is found to have an inconsistent problem after the system is restarted, and it needs to be repaired by fsck or scandisk, which is very time-consuming and may not be able to be repaired successfully in the end. To make matters worse, if this is a server that needs to do fsck, it can only be offline (offline), and the existing applications are often large hard drives, and the corresponding fsck repair time is also very long, which is almost intolerable for many users using the server.

After using ZFS, you can abandon fsck completely, because ZFS is a file system based on COW (Copy on Write) mechanism. COW does not rewrite existing files on the hard disk, ensuring that all files on the hard disk are valid. So there is no such concept of inconsistent, naturally there is no need for this kind of tool.

two。 Easy to manage

As a brand-new file system, ZFS completely abandons the traditional File System + Volume Manager + Storage architecture. All storage devices are managed through ZFS Pool. As long as all kinds of storage devices are added to the same ZFS Pool, everyone can easily manage the configuration file system in this ZFS Pool. You no longer have to keep in mind all kinds of professional concepts, various commands newfs, metinit and the usage of various Volume Manager. In ZFS, we only need two commands, zpool (needle-to-ZFS Pool management) and zfs (for ZFS file system management), to easily manage 128bit file systems. For example, we often encounter that the system data is growing too fast, the existing storage capacity is not enough, and we need to add a hard disk. If we follow the traditional Volume Manager management method, we need to consider many existing factors in advance and calculate various parameters that need to be configured according to the application in advance. In the case of ZFS, our system administrator can be completely liberated, no longer need this kind of artificial complex consideration and calculation, we can leave these to ZFS, because ZFS Pool will automatically adjust to dynamically adapt to the needs. All we need is a simple command to add a new hard drive to the ZFS Pool:

Zpool add zfs_pool mirror c4t0d0 c5t0d0

Based on this dynamically adjusted ZFS Pool, all file systems can be used immediately on this new hard disk, and the optimized parameters will be automatically selected. And ZFS also provides a graphical management interface.

3. There is no capacity limit

The ZFS (Zettabyte File System) file system, as its name suggests, can provide truly massive storage, making it almost impossible to encounter capacity problems in reality. Under the existing 64-bit kernel (kernel), it can hold a single file up to 16 Exabytes (264) size, can use 264 storage devices, and can create 264 file systems.

4. Fully guarantee the correctness and integrity of the data

Because all the data operations in ZFS are based on transaction, a set of corresponding operations will be parsed into a transaction operation by ZFS, and the transaction operation represents that a group of operations either fail or succeed together. And as mentioned earlier, ZFS is based on COW (Copy on Write) for all operations, thus ensuring that the data on the device is always valid and will no longer cause inconsistent of data files due to system crashes or accidental power outages.

Another potential threat data may come from hardware problems, such as magnetic disks, raid card hardware problems, or driver bug. Existing file systems often encounter this problem, often simply send the wrong data directly to the upper application, usually we call this problem Silent Data Corruption. In ZFS, a 256bit Checksum is performed on all data, whether it is user data or the metadata data of the file system itself. When ZFS submits the data, it will be verified, which completely eliminates this Silent Data Corruption situation.

5. Provide excellent performance and scalability

Different from the traditional File System + Volume Manager + Storage architecture, ZFS provides all the functions directly based on the storage device, so it has its own unique innovative characteristics, and the performance is naturally extraordinary.

Dynamic Striping vs. Static Striping

Because ZFS is based on COW and a global dynamic ZFS Pool, any write operation is a write operation to a new block of data (Block). ZFS dynamically picks out an optimal device from ZFS Pool and writes with a transaction (business) linear, which makes full use of the bandwidth of existing devices. We call this feature Dynamic Striping. The corresponding Static Striping is the way used by the traditional file system, Static Striping requires the administrator to correctly calculate the artificial settings for this group of Stripe in advance, and if you add new devices, you need to re-artificial calculation and setting, what is more serious is that if the human calculation error, it will directly affect the performance of the system. After using the feature of Dynamic Striping, we don't need human intervention at all. ZFS will adjust automatically and intelligently provide you with the best device and the fastest mode of operation.

Support for blocks of multiple sizes (Multiple Block Size)

ZFS supports block definitions of various sizes, from 512 bytes to 1m bytes. Unlike traditional file systems, which are often fixed size data blocks, ZFS can dynamically calculate according to different size files and dynamically select the best data block.

Because of the different size of data blocks, it directly affects the actual hard disk capacity and reading speed. If you use smaller data blocks, storing files results in less fragmentation and it is faster to read and write small files, but it results in the need to create more metadata, and it takes more time to read and write large files. If you use larger data blocks, you use less metadata, which makes it easier to read and write large files, but results in more fragmentation. According to the actual investigation of the use of existing files, ZFS analyzes an algorithm for selecting data block size, and dynamically determines the best data block according to the actual file size. So ZFS is very smart and can get a self-tuning result without the intervention of the system administrator. Of course, ZFS also supports custom settings for the block size used by a single file or the entire file system.

Intelligent pre-reading (Intelligent Prefetch)

Most operating systems have this ability to read data in advance, while ZFS is a more intelligent data pre-reading function that is provided directly on the file system. It can not only intelligently identify a variety of reading modes, read data in advance, but also carry out this pre-reading intelligent identification for each read data stream, which is a very good thing for many streaming media providers.

In terms of scalability, unlike most of the existing file systems, which are based on a limited static model, ZFS adopts the dynamic concept of ZFS Pool, its metadata is also dynamic, and its read and write operations are parallel and have the concept of priority, so it can guarantee the linear growth of performance even in the case of large amount of data and multiple devices.

6. Self-repairing function

ZFS Mirror and RAID-Z

The traditional hard disk Mirror and RAID 4 Grand raid 5 array will encounter the aforementioned problem: Silent Data Corruption. If a physical problem occurs on a hard disk and causes a data error, the existing Mirror, including the RAID 4e raid 5 array, will silently submit the error data to the upper application. If this error occurs in Metadata, it will directly result in the Panic of the system. And there is a more serious situation: in RAID 4 and RAID 5 arrays, if a power outage occurs when the system is calculating Parity values and writing new data and new Parity values again, then all the data stored in the entire array is meaningless.

In ZFS, the corresponding ZFS Mirror and RAID-Z methods are proposed. When it is responsible for reading data, it will automatically check with the 256bit check code, actively find the Silent Data Corruption, and then get the correct data back to the upper application through the corresponding Mirror hard disk or other hard disks in the RAID-Z array, and automatically repair the Data Corruption of the original hard disk at the same time.

Fault Manager

In Solaris 10, a ZFS diagnostic engine interacts with Solaris's Fault Manager (another new feature of Solaris 10) to diagnose, analyze and report errors in ZFS Pool and storage devices in real time, and users can get a very friendly message in time through Fault Manager. Although the diagnostic engine does not take proactive actions to fix or resolve the problem, it prompts the system administrator for actions that can be taken in the message. Similar to the following ZFS error message, where REC-ACTION is the recommended action:

SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: MajorEVENT-TIME: Fri Mar 10 11:09:06 MST 2006PLATFORM: SUNW,Ultra-60, CSN: -, HOSTNAME: neoSOURCE: zfs-diagnosis, REV: 1.0EVENT-ID: b55ee13b-cd74-4dff-8aff-ad575c372ef8DESC: A ZFS device failed. Refer to http://sun.com/msg/ZFS-8000-D3 for more information.AUTO-RESPONSE: No automated response will occur.IMPACT: Fault tolerance of the pool maybe compromised.REC-ACTION: Run 'zpool status-x' and replace the bad device.

7. Safety

In terms of security, ZFS supports ACL (read control list) similar to the NT-style NFSv4 version. And the aforementioned 256bit CAPTCHA code, users can choose a variety of authentication methods, including SHA-256 authentication algorithm, so as to ensure the security of data at the physical storage unit level.

8. Super function

As the "last file system", ZFS covers the basic file system and Volume management functions, while providing many enterprise-level super functions: Quota (quota), Reservation (reservation), Compression (compression), Snapshot (snapshot), Clone (clone). And very fast. With this file system, you don't need any Volume Manager anymore.

9. Compatibility

ZFS is a file system that is fully compliant with the POSIX specification, so the upper application is completely unaffected. ZFS also provides an Emulated Volume module that can use any ZFS file system as a normal block device. At the same time, ZFS can also use the Volume built on Volume Manager as the storage device unit. In this way, without the need to modify the application and the existing file system, it gives everyone the maximum freedom to obtain the various features provided by ZFS.

10. Open source

ZFS is a Sun Microsystems company operating as an open source project of OpenSolaris and is completely free to use, which means that we can not only enjoy the high quality of commercial companies, but also gain the advantages of the open source model.

Although currently only Solaris supports this file system, this open source model is bound to promote more ZFS-based applications. Now some foreign developers are porting ZFS to Linux and Mac OS. If you want to experience ZFS, because it is currently bundled with Solaris 10, you need to download the latest version of Solaris 106 Universe 06 (http://www.sun.com/software/solaris/get.jsp).

Additional instructions

This article discusses the advantages of ZFS. Now, let me tell you a very realistic question about ZFS. Using RAID-Z can be expensive because you need to buy a lot of disks to increase storage space.

Have you used ZFS yet? What is your experience in using it?

What is Linux system Linux is a free-to-use and free-spread UNIX-like operating system, is a POSIX-based multi-user, multi-task, multi-threaded and multi-CPU operating system, using Linux can run major Unix tools, applications and network protocols.

Thank you for reading! This is the end of the article on "what does ZFS in linux refer to?". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, you can share it for more people to see!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.