Shulou(Shulou.com)06/02 Report--

Arrange for an engineer to install Oracle on a production server, and the engineer installs it while studying it. It feels wrong and is ready to uninstall and reinstall. Find the uninstall method on the Internet, where you need to execute a single command to delete the installation directory of Oracle, as follows:

Rm-rf $ORACLE_BASE/*

If the variable ORACLE_BASE is not assigned, the command becomes

Rm-rf / *

The girl is using a root account. In this way, all the files of the whole disk are deleted, including the application of Tomcat and MySQL database and so on.

Isn't the mysql database running? can linux delete executing files? Anyway, it was deleted completely, and in the end, there was only one log file for tomcat. It is estimated that the file is too large to be deleted successfully.

Looking at the self-remorse look in the girl's eyes, it was because I arranged for her to do it, and I didn't make it clear to her. Without any training, I had to bear the responsibility alone. Moreover, how could I let the beautiful woman bear this responsibility?

Call the computer room, hang the disk to another server, ssh to see that all the files have been cleared, this server is running a customer's production system, ah, has been running for more than half a year, it has to be restored as soon as possible. So find the offline backup database, found that the backup file is only 1kb, there are only a few lines of familiar mysqldump comments (is there something wrong with the backup script executed by crontab), and the last backup is from December 2013.

Think of a case that a leader said: when a production system hung up, it was found that all the backups were problematic, the burned CDs were scratched, and the tape drive was broken (a senior in the industry, probably used CDs for backup in the past). I didn't expect that it really happened to me today, what should I do?

After knowing the situation, the department leader has made the worst plan B: the leader personally led the team and the product AA to the customer's local city on Sunday and went to the leadership to communicate on Monday; BB and CC went to the customer administrator to find a way to convince the customer.

Ext3grep, a lifesaver

Quickly go to the Internet to look up information for mistakenly deleted data recovery, and really find an ext3grep that can recover files deleted through rm-rf. Our disk is also in ext3 format, and there are many successful cases on the Internet. So lit up a glimmer of hope, quickly to the disk umount, to prevent rewriting add and delete file sector. Download ext3grep and install (the compilation and installation process is arduous for the time being).

Execute the scan file name command first:

Ext3grep / dev/vgdata/LogVol00-- dump-names

Printed out all the deleted files and paths, ecstatic, do not have to implement Plan B, the files are here.

This software cannot restore files by directory, but can only execute the restore command:

Ext3grep / dev/vgdata/LogVol00-- restore-all

As a result, the current disk space is insufficient, so we have no choice but to recover the files. I tried several files, but some of them succeeded and some failed.

Ext3grep / dev/vgdata/LogVol00-- restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

I can't help but feel cold in my heart. Is it possible to delete the file that has been written on the disk? The probability of recovery is not big, ah, can restore a few count, perhaps important data files happen to be in the recoverable MYD files. So redirect all file names to a file first.

Ext3grep / dev/vgdata/LogVol00-- dump-names > / usr/allnames.txt

Filter out the file names of all mysql databases, mysqltbname.txt

Write a script to restore files:

While read LINEdo echo "begin to restore file" $LINE ext3grep / dev/vgdata/LogVol00-- restore-file $LINE if [$?! = 0] then echo "restore failed, exit" # exit 1 fidone <. / mysqltbname.txt

Execution, about 20 minutes to run, recovered more than 40 files, but not enough, we nearly 100 tables, each table frm,myd,myi three files, how to say there are more than 300 ah! Attach the recovered files to the existing database, and even after the file permission is 777, restart mysql, which can be regarded as part of the data recovery, but the customer's important attendance check-in data and mobile phone report data (it is said that customers use these data to do employee performance) have not yet been found.

What should I do? In the middle, I tried another tool, extundelete, which is basically the same as ext3grep syntax, and the principle should be the same, but it is said that it can be restored by directory. All right, give it a try.

Extundelete / dev/vgdata/LogVol00-- restore-directory var/lib/mysql/aqsh

As expected, I can't recover! Those documents have been destroyed. Report to the leader and implement Plan B. Have no choice but to go home from work (weekend, go back to have a rest, think of a way)

Brainstorm: binlog

I woke up early the next morning (something on my mind), carried my computer and went to the company (this weekend is reimbursed, free of criticism, notice, fine, dismissal, what weekend?).

Still running ext3grep,extundelete, just a few tricks, put the system on the test server and see if the data can find a way to fix it. Mysqldump on the test server, restore the files, overwrite the recovered files, add permissions to the files, and restart mysql.

Wait,wait, isn't there a binlog? All of our services are required to enable binlog. Maybe we can recover data through binlog?

So from the dump file name to find binlog files, a total of three, mysql-binlog0001,mysql-bin.000009,mysql-bin.000010, restore 0001

Ext3grep / dev/vgdata/LogVol00-- restore-file var/lib/mysql/mysql-bin.000001

I can't believe I failed.

Look at the other two files, mysql-bin.000010 about a few hundred MB, should be more reliable, the implementation of the restore command, unexpectedly successful!

Quickly scp to the test server. Perform a binlog restore.

Mysqlbinlog / usr/mysql-bin.000010 | mysql- uroot-p

Enter the password, it is stuck (good sign), after a long wait, it is finally over. Open the app, oh, thanks to cctv,mtv, the data is back!

Postscript

After the accident, although the data were lucky to be found, the process was shocking. They are also afraid of the consequences of their own mistakes and the joint and several liability they bring to colleagues and leaders. I also hope to remember this accident and not make the same mistake in the future. The reflection on the accident is as follows:

1. When arranging MM for server maintenance, I didn't explain the situation to her in advance, and I didn't pay attention to it either. The management was chaotic and the process was chaotic. In an online production system, any change must be planned first and then moved later.

two。 There is a problem with the automatic backup and no one is checking it. Offline backup staff never pay attention to the 1k files downloaded from the server each time. You need to be clear about your responsibilities at work.

3. After the accident, it was not found in time, resulting in part of the data written to disk, resulting in unrecoverable problems. You need to write an application monitoring program, and if there is an exception in the service, SMS will alert the relevant responsible person.

According to the comment reminder, add one more:

4. You cannot use the root user to operate. Users with different permission levels should be set up on the server.

Through this accident, several colleagues who had nothing to do with the project and the accident offered to help, check the information, and help with the test, and one colleague even helped to conduct the data recovery test at more than 1: 00 p. M. At the same time, when the product manager thinks of the great pressure on the customer, he does not panic and blame the developer and the specific operator, but allows everyone to calm down and think about the solution. The department leaders also take the initiative to help think of ways to accompany us to work overtime tests and track the progress of things in real time.

Through everyone's joint efforts, things finally came to a relatively successful end. next, we will conduct a collective reflection on Monday morning and sum up experiences and lessons, and we must try our best to avoid such accidents.

/ * Portal * /

Links to the tools used in this article:

1.ext3grep: https://code.google.com/p/ext3grep/

Compilation and installation depends on more packages, you can go to the Internet to search how to install. Unfortunately, the howto given by the author has been blocked. I FQ downloaded the pdf document of howto. After reading it, you will have a better understanding of linux's file system.

This tool has a bug, and ext3grep: init_directories.cc:534: void init_directories (): Assertion `lost_plus_found_directory_iter! = all_directories.end () 'failed., will not be executed downwards after an error, resulting in recovery failure. The author released a patch at the download address: patch download. I don't understand why the author didn't include this patch in the new edition.

2.extundelete: http://extundelete.sourceforge.net/

The function is similar to that of ext3grep, and the principle should be similar. Only claimed to be able to restore the directory, I did not succeed here.

Postscript: now more and more big companies disable rm. If you want to remove a file, they will choose mv to a junk file and deal with it centrally at regular intervals. Avoid many embarrassing faces caused by erroneous deletion!

Summary

The above is the recovery process of mistakenly deleted files on the recording server rm-rf introduced by the editor. I hope it will be helpful to you. If you have any questions, please leave me a message and the editor will reply to you in time. Thank you very much for your support to the website!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.