Shulou(Shulou.com)06/02 Report--

[user information] A company in Shanghai

[fault description] the device that needs data recovery is an EMC CX4 storage server, because the failure of the hard disk paralyzes the entire storage array. The whole LUN is a RAID 5 made up of seven 1TB hard drives. But the server has a total of 10 hard drives, of which 3 are disconnected from previous storage. (the hard drive was dropped during the use of the server, but the customer only added a new hard disk to do rebuild, and did not unplug the disconnected hard drive, so there were 3 more.)

Because the customer said that some of the hard drives in the storage server were malfunctioning, the storage server was unavailable, so the preliminary judgment is that there is a hard disk in the RAID that is offline. Give all the hard drives to the hardware engineer to check if there is any hardware failure. After the test is completed, it is found that there is no physical failure, then there may be a logical bad path on the hard disk or other reasons that make the RAID unavailable. Then all the hard drives are fully mirrored to ensure the security of the customer's existing data.

[recovery process] after mirroring, I began to analyze the structure of the RAID. After opening the mirror disk, I found that the hard disk added an extra 8-byte check for every 512 bytes, that is, it became 520 bytes per sector. Although you can analyze the structure of RAID, it will be very difficult to reorganize RAID! In order to improve work efficiency, we need to write a Mini Program to remove the 8-byte check to facilitate later work. After all the disks have been converted, the structure of the RAID is analyzed. Due to the addition of 3 old disks, we need to compare each disk, that is, the front part of the two disks will be the same, and one of the two disks will be old, and the amount of data of the old disk is not as large as that of the new disk, so the old disk can be excluded. There will be three pairs of such disks, so you can rule out all the old disks. Next, let's take a look at the RAID structure. Because of the NTFS file system used by customers, it is easy to find the structure of RAID with MFT. Knowing the RAID structure, I found that this is not an ordinary RAID 5, but a double loop. There is no way to reassemble RAID with winhex. Finally, it is found that UFS can reorganize RAID. Then use it to reorganize RAID first! After reorganizing RAID, it is found that the data is not up to date. I think it may be that RAID 5 dropped a hard disk first, but the administrator did not find it in time and did not add a new hard disk in time to do rebuild. As a result, another hard drive goes offline after running for a period of time, causing the entire RAID to be unavailable. So you also need to find an old disk to generate the latest data. Continue to find the disk! But how to find it? We try to kick out each hard drive in turn, and then reassemble RAID. See if the latest data is available. Until you find the latest data available, you can then export the data and validate it.

[recovery results] the whole recovery process, including mirroring, sector conversion and final copying of data, takes a total of 5 days. The data recovery rate is more than 99%. Although the whole process is relatively long, but the end result users are very happy, because it brought them unnecessary losses.

Author: Deng Qi (Server data recovery engineer)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.