Shulou(Shulou.com)06/02 Report--

Server failure description:

EMC FC AX-4 storage RAID5 disk array of a server of a company in Shanxi Province. There are 12 hard disks in the array to form a raid5 disk array. Two hard disks are hot spares. The capacity of a single hard disk in the array is 1TB. Two hard disks in the server are offline, and one hot spare is not enabled. The customer brings all the disks in the server to the data recovery company.

Generally, the reason for server hard disk offline is disk physical failure or hard disk bad sectors. However, since EMC controller has a very strict disk inspection policy, it is easy to determine the unstable hard disk performance as hardware failure and put forward raid group, so the cause of server crash may also be unstable disk reading and writing.

Server Data Recovery Resolution Process:

Step 1: Check the hard disk and server data backup; check the physical failure of all disks in the server, there is no physical failure of the hard disk, and then use the bad track Detection Tools to check the bad track of the hard disk. Everything is normal. Use a professional mirroring tool to mirror all disks in the raid. As shown below:

Step 2: Analyze the RAID group structure; the normal steps for Raid data recovery are to analyze the server raid information and then reconstruct the raid group. In this case, it is found that disk 6 and disk 9, which are hot spare disks, have no data. Disk 6 has been successfully activated and replaced disk 5 in the disk array, but the data is not synchronized. Continue to analyze other hard disks in the server raid for necessary information such as stripe size, data distribution law, disk order, etc. The analysis found that the data of No.7 hard disk in the same strip is different from other hard disks in the raid. It is preliminarily confirmed that the disk is the hard disk that dropped earlier. The data recovery company's own raid verification program is used to verify this strip. It is found that the best data is the data after removing No.7 disk, so No.7 disk is undoubtedly the first to drop. the analyzed information is used to construct the original raid disk array through raid virtual program independently developed by north asia.

Step 3: Analyze the LUN information in the server disk array; only one LUN is allocated at the bottom of the server, so the workload is relatively small. You only need to analyze the information of one LUN, and use the raid recovery program to interpret the map data and export the memory after analysis. Then use your own software for zfs file system interpretation, some file system files in the parse error. Engineers had to debug the program manually and found that the error was caused by the sudden collapse of the server, which caused some metafiles to be damaged. The existing program could not be explained normally. Therefore, these corrupted file system metafiles need to be repaired in order to properly parse ZFS file systems. Analyzing the damaged metafile, it is found that some file system metafiles are not updated and damaged due to the storage paralysis while the ZFS file is undergoing IO operations. Manual repair of these damaged metafiles ensures that ZFS file systems can be parsed properly.

Step 4: Export all successfully recovered data; use the program to parse the repaired ZFS file system, parse all file nodes and directory structures. Verify that all successfully recovered data is complete. Some file directories and validation screenshots are as follows:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.