Shulou(Shulou.com)06/01 Report--

This article mainly introduces the centos log file system example analysis, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1. Log file system

Usually, when the contents of the file are written while the system is running, the metadata of the file (such as permissions, owner, and creation and access time) is not written. If the system shuts down abnormally and the file system in the writing process will be unmounted abnormally, then the file system will be in an inconsistent state if the time difference between writing the file content and writing the file metadata. When restarted, Linux runs the fsck program, scans the entire file system, ensures that all file blocks are allocated or used correctly, finds the corrupted directory entry, and tries to repair it. However, fsck does not guarantee that the damage will be repaired. When this happens, inconsistent metadata in the file will fill the space of the lost file, and the file entry in the directory entry may be lost, resulting in the loss of the file.

In order to minimize the inconsistency of the file system and shorten the startup time of the operating system, the file system needs to track the records that cause system changes, which are stored in a place separate from the file system, which is usually called "log". Once these log records are safely written, the log file system can apply them to clear the records that cause changes in the system, form them into a set that causes changes in the file system, and put them in the transaction of the database. keep the valid data running normally in the state without conflicting with the performance of the whole system. When any system crashes or needs to be restarted, the data is restored in accordance with the information recorded in the log file. Because there are periodic checkpoints in the log file, it is usually very neat. The design of the file system is mainly concerned with efficiency and performance.

Linux can support many journaling file systems, including FAT, VFAT, HPFS (OS/2), NTFS (Windows NT), UFS, XFS, JFS, ReiserFS, ext2, ext3, and so on.

2. Advantages of ext3

Why do you need to migrate from ext2 to ext3? There are four main reasons: availability, data integrity, speed, and ease of migration.

Usability

After an abnormal crash (power outage, system crash), the ext2 file system can be mounted and used only after the consistency check is done through e2fsck. The time it takes to run e2fsck depends largely on the size of the ext2 file system. It takes a long time to verify a slightly larger file system (dozens of GB). If there are a large number of files on the file system, the verification time is longer. It may take an hour or more to validate a file system with hundreds of GB. This greatly limits availability. In contrast, ext3 does not require file system verification unless there is a hardware failure, even if the system shuts down abnormally. This is because the data is written to disk in a way that the file system is always consistent. After an abnormal shutdown, the time it takes to restore the ext3 file system does not depend on the size of the file system or the number of files, but on the size of the "logs" required to maintain consistency. With the default log settings, the recovery time is only one second (depending on the hardware speed).

Data integrity

With the ext3 file system, the data integrity performance is reliably guaranteed when the computer is shut down abnormally. You can choose the type and level of data protection. You can choose to keep the file system consistent, but allow the data on the file system to be damaged during an abnormal shutdown; this can improve some speed in some cases (but not all). You can also choose to keep the reliability of the data consistent with the file system; this means that you won't see any data junk in the newly written files after the crash. This secure choice that keeps the reliability of the data consistent with the file system is the default.

Speed

Although ext3 writes more data than ext2, ext3 is often faster than ext2 (high data stream). This is because the log function of ext3 optimizes the rotation of the hard disk head. You can choose one of the three logging modes to optimize speed, selectively sacrificing some data integrity. The first mode, data=writeback, guarantees data integrity to a limited extent, allowing old data to exist in files after a crash. This mode can increase speed in some cases. (in most journaling file systems, this mode is the default. This mode provides limited data integrity for the ext2 file system, and is more likely to avoid long file system validations when the system boots.) the second mode, data = orderd (the default mode), keeps the reliability of the data consistent with the file system; this means that you won't see any junk data in newly written files after a crash. The third mode, data=journal, requires larger logs to ensure moderate speed in most cases. It also takes a long time to recover after the crash. But it will be faster in some database operations. In general, the default mode is recommended. If you need to change the mode, add the data= mode option to the appropriate file system in the / etc/fstab file. For more information, see the man page online manual of the mount command (execute man mount).

Easy to migrate

You can enjoy the benefits of a reliable journaling file system without reformatting your hard drive and easily migrating from ext2 to ext3. Yes, you don't need to do a long, boring, possibly wrong "backup-reformat-restore" operation to experience the advantages of ext3. There are two ways to migrate:

If you upgrade your system, the Red Hat Linux installer will assist with the migration. All you need to do is press the select button for each file system.

Using the tune2fs program, you can add logging to an existing ext2 file system. If the file system has been mounted during the conversion process (mount), the text ".journal" will appear in the root directory; if the file system is not mounted, the file will not appear in the file system. To convert a file system, just run tune2fs-j / dev/hda1 (or any device name of the file system you want to convert) and change the ext2 in the file / etc/fstab to ext3. If you want to convert your root text system, you must boot using initrd. Follow the mkinitrd manual to run the program and verify that your LILO or GRUB configuration is loaded with initrd (if not successful, the system will still boot, but the root file system will be mounted as ext2, not ext3, as you can confirm with the command cat / proc/mounts. For more information, see the man page online manual of the tune2fs command (execute man tune2fs).

3. Three log modes of ext3

Ext3 provides a variety of logging modes, that is, the ext3 file system can support either changing the metadata of the file system or changing the data of the file system (including changes in the file itself). Here are three different logging modes that are activated when the / etc/fstab file boots:

Data=journal log mode

The log records include all data and metadata that change the file system. It is the slowest of the three ext3 logging modes, but it minimizes the possibility of errors. Using "data=journal" mode requires ext3 to write each change to the file system twice and log once, which reduces the overall performance of the file system. All new data is first written to the log before it is located. After an accident, the log can be replayed to bring the data back to a consistent state with the metadata. Because metadata and data updates are recorded in ext3, these logs will work when a system is restarted.

Data=ordered log mode (default)

Only the metadata that changes the file system is recorded, and the overflow file data is added to disk. This is the default ext3 logging mode. This mode reduces redundancy between writing to the file system and writing to the log, so it is faster, although changes in file data are not recorded in the log, they must be done, and are executed by ext3's daemon program before the associated file system metadata changes, that is, to modify the file system data before recording the metadata, which will slightly slow down the system performance (speed) However, you can ensure that the file data in the file system is synchronized with the metadata of the corresponding file system.

Data=writeback log mode

Only the metadata that changes the file system is recorded, but according to the standard file system, the writer still records the changes in file data on disk to maintain file system consistency. This is the fastest ext3 logging mode. Because it only records the changes of metadata, and does not need to wait for updates related to file data, such as file size, directory information, etc., the update of file data and record metadata changes can be out of sync, that is, ext3 is a log that supports asynchronous. The defect is that when the system is shut down, the updated data is in contradiction because it cannot be written to disk, which is not well resolved at present.

There are differences between different logging modes, but the setting method is just as convenient. You can use the ext3 file system to specify the logging mode, which is done by / etc/fstab at startup. For example, if you select data=writeback log mode, you can make the following settings:

/ dev/hda5 / opt ext3 data=writeback 1 0

In general, data=ordered logging mode is the default mode for ext3 file systems.

To specify the logging method, you can use the following methods:

1 add the appropriate string such as data=journal to the option field of / etc/fstab

# / dev/sda3 / var ext3 defaults,data=writeback 1 2

2 specify the-o data=journal command line option directly when calling mount.

# mount-o data=journal / dev/sdb1 / mnt

If we want to see how the log of a file system should be queried, we can use the dmesg command here:

# dmesg | grep-B1 "mounted filesystem"

Kjournald starting. Commit interval 5 seconds

EXT3-fs: mounted filesystem with ordered data mode.

--

EXT3 FS on sda1, internal journal

EXT3-fs: mounted filesystem with ordered data mode.

--

EXT3 FS on sdb1, internal journal

EXT3-fs: mounted filesystem with journal data mode.

--

EXT3 FS on sdb1, internal journal

EXT3-fs: mounted filesystem with writeback data mode.

4. Select log mode

Speed

In some typical cases, using the option data=writeback can significantly increase speed, but at the same time reduce the protection of data consistency. In these cases, the protection of data consistency is basically the same as that of the ext2 file system, except that the system continuously maintains the integrity of the file system during normal operation (this is the logging mode used by other journaling file systems). This includes frequent shared writes, as well as frequent creation and deletion of a large number of small files, such as sending a large number of small email messages. If you switch from ext2 to ext3 and notice a significant decline in application performance, the option data=writeback may help you improve performance. Even if you don't get expensive data consistency protection, you can still enjoy the benefits of ext3 (file systems are always consistent). Red Hat is still working to improve some aspects of ext3 performance, so some aspects of ext3 performance can be improved in the future. This also means that even if you choose data=writeback now, you will need to retest the future version with the default value of data=journal to determine whether the changes to the new version are related to your work.

Data integrity

In most cases, the user writes data at the end of the file. Only in some cases, such as a database, the user writes data in the middle of an existing file. Even overwriting an existing file is achieved by truncating the file and then writing data from the end of the file. In data=ordered mode, if the system crashes while a file is being written, the data block may be partially overwritten, but the writing process is not completed, so there are incomplete data blocks that do not belong to any file. In data=ordered mode, the only situation in which unordered blocks remain after a crash is when a program is rewriting a file during the crash. In this case, the write order cannot be guaranteed absolutely unless the program uses fsync () and O_SYNC to force write operations in a specific order.

The ext3 file system also involves how the data in cache is brushed to the hard disk. It is brushed regularly through the kupdate process, which is checked every 5 seconds by default and brushes more than 30 seconds of dirty data to the hard disk.

This can be achieved by modifying / proc/sys/vm/bdflush in as 3.0. In as 4.0, you can do this by modifying / proc/sys/vm/dirty_writeback_centisecs and / proc/sys/vm/dirty_expire_centisecs.

Because the default is ordered mode, in this mode, if an IO writes the data file first and then writes the log file. If crashed occurs in the system after writing the data file and before writing the log file, then this part of the data will be lost, which is absolutely not allowed in the database, whether it is Oracle or MySQL. So for database writes, each write operation will first be written to pagecache, and then tell kernelthread to brush the buffers to the hard disk, then write the metadata to the log, and finally return the successful write operation. In this way, the write operation for the database is obviously not as fast as the writing device.

So in the case of using Ext3 to run the database, setting the logging mode to journal mode should improve the performance instead (not tested, in theory). Because a write operation of the database in journal mode first writes the data and file system changes directly to the log (bypassing cache for better performance), then writes the data to cache, and then refreshes the data to the hard disk by the kupdate process. In contrast, for DB, its performance should be faster than the previous one.

In addition, we also mention the parameter sync_binlog in MySQL. If you set this parameter to 1, that means that each time you write the binlog file, it will be brushed to the hard disk at the same time, just like writing IO in Oracle. If this parameter is turned off, it will be managed by OS, that is, it will be checked every 5 seconds and found that 30 seconds old data will be brushed to the hard disk. The innodb_flush_log_at_trx_commit parameter also involves the problem of brushing the hard disk.

As an enhanced version of ext2, ext3 is almost identical to the superblock, inode, group descriptor and other data structures used by ext2, so ext3 is forward compatible with ext2. Without backing up ext2 file system data, you can use:

one

# tune2fs-j/dev/sd1

Convert the ext2 file system directly to the ext3 file system without unmounting the partition.

What would happen if we had a power outage or the system was locked and forced to restart while we were editing the file? Some of the contents of the file are lost, the contents of the whole file are confused, and even the file system crashes directly. What a terrible thing it will be. When linux shuts down normally, we all see a print message that unmounts the file system, and the abnormal shutdown will cause the file system to be inconsistent, which will be found when the file system is suspended during the system reboot phase, and then it will try to fix it. Unfortunately, as the capacity of the storage device increases, the time spent on this repair becomes more and more intolerable.

The biggest feature of Ext3 is the addition of logging function to ext2, so the ext3 file system is often called journal file system, but the journal file system is not only ext3, but also such as JFS, reiserFS and XFS, as well as NTFS that we often see on Windows.

The logging feature of Ext3 is mainly accomplished by an intermediate device called "log block device layer" in its lower layer, called JBD (Journaling Block Device layer referred to as JBD). JBD is not part of the file system specification, it has nothing to do with the specification of the ext3 file system, and JBD is the basis for the implementation of the file system transaction function. In short, JBD is designed to achieve the special purpose of logging on any block device (more and more abstract, what is a transaction? ⊙ transactions ⊙... (.)

With regard to transactions, students who have experience in database development or data operation and maintenance will certainly be no stranger. Here we not only conclude the concept, but also do not adhere to the academic definition, as long as we know that the main function of the transaction is to ensure the atomicity of the operation. How do you understand this sentence? For example, in the financial system, X yuan is transferred from account A to account B. This business must ensure that X yuan is successfully transferred from An account, and then X yuan is successfully added to B account. Only if these two operations are successful at the same time can the task be a successful transfer. If either operation fails, the business must be terminated. If the transfer of X yuan from An account is successful and there is an error when writing to B account, then the X yuan transferred from An account must be returned to An account. Even more extreme, at this time, the data of An account crashes again for various reasons, so the transaction mechanism of the database must ensure that the X yuan of An account will not be lost. This is the atomicity of database business operations. In the log file system, the atomicity of the operation on file data is guaranteed by JBD, and Ext3 realizes its logging function through the API of "hooking in" JBD. Although the JBD layer itself does not have much code, it is a very complex part of the software. Let's ignore it here and play with it later.

Of course, the log file system records logs, and logs also need to take up storage space. Therefore, the log file system is to open up a special area on the storage media to store log information:

We use a picture to briefly describe the underlying layout of ext3:

Thank you for reading this article carefully. I hope the article "sample Analysis of Log File system in centos" shared by the editor will be helpful to you. At the same time, I also hope you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.