Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to solve the rm-rf problem of Oracle database". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to solve the rm-rf problem of Oracle database.

Accident background

Arrange for a girl to install Oracle on a production server. She studies and installs it. She feels wrong and is ready to uninstall and reinstall.

Find the uninstall method on the Internet, where you need to execute a single command to delete the installation directory of Oracle, as follows:

Rm-rf $ORACLE_BASE/*

If the variable ORACLE_BASE is not assigned, the command becomes:

Rm-rf / *

Wait, wait. the girl uses a Root account. In this way, all the files of the whole disk are deleted, including the application of Tomcat and MySQL database and so on.

Isn't the MySQL database running? Can Linux delete files that are in progress? Anyway, it was deleted completely, and finally there was a Log file for Tomcat. It was estimated that the file was too large to be deleted successfully for a while.

Looking at the self-remorse look in the girl's eyes, it was because I arranged for her to do it, and I didn't make it clear to her. Without any training, I had to bear the responsibility alone. Moreover, how could I let the beautiful woman bear this responsibility?

Call the computer room, hang the disk to another server, SSH to see that all the files have been cleared, this server is running a customer's production system, ah, has been running for more than half a year, it has to be restored as soon as possible.

So find the offline backup database, found that the backup file is only 1KB, there are only a few lines of familiar mysqldump comments (is there something wrong with the backup script executed by Crontab), and the closest backup is from December 2013.

Think of a case that a leader once said: when a production system hung up, it was found that all the backups were problematic, the burned CDs were scratched, and the tape drive was also broken (a senior in the industry, probably used CDs for backup in the past). I didn't expect that it really happened to me today, what should I do?

After knowing the situation, the department leader has made the worst plan B: the leader personally led the team and the product AA to the customer's local city on Sunday and went to the leadership to communicate on Monday; BB and CC went to the customer administrator to find a way to convince the customer.

Straws: ext3grep

Quickly go to the Internet to look up information for mistakenly deleted data recovery, and really find an ext3grep that can recover files deleted through rm-rf. Our disk is also in ext3 format, and there are many successful cases on the Internet.

So lit up a glimmer of hope, quickly umount the disk to prevent rewriting add and delete file sector. Download ext3grep and install (the compilation and installation process is arduous for the time being).

Execute the scan file name command first:

Ext3grep / dev/vgdata/LogVol00-- dump-names

Printed out all the deleted files and paths, ecstatic, do not have to implement Plan B, the files are here.

This software cannot restore files by directory, but can only execute the restore command:

Ext3grep / dev/vgdata/LogVol00-- restore-all

As a result, the current disk space is insufficient, so we have no choice but to recover files. Several files were tried, but some of them were successful and some failed:

Ext3grep / dev/vgdata/LogVol00-- restore-file var/lib/mysql/aqsh/tb_b_attench.MYD

I can't help but feel cold in my heart. Is it possible to delete the file that has been written on the disk? The probability of recovery is not big, ah, can restore a few count, perhaps important data files happen to be in the recoverable MYD files.

So redirect all file names to one file first:

Ext3grep / dev/vgdata/LogVol00-- dump-names > / usr/allnames.txt

Filter out the file names of all MySQL databases and save them as mysqltbname.txt.

Write a script to restore files:

While read LINEdo echo "begin to restore file" $LINE ext3grep / dev/vgdata/LogVol00-- restore-file $LINE if [$?! = 0] then echo "restore failed, exit" # exit 1 fidone <. / mysqltbname.txt

Execution, about 20 minutes to run, recovered more than 40 files, but not enough, we nearly 100 tables, each table frm,myd,myi three files, how to say there are more than 300 ah!

Attach the recovered files to the existing database, and even after the file permission is 777, restart MySQL, which can be regarded as part of the data recovery, but the customer's important attendance check-in data and mobile phone report data (it is said that customers use these data to do employee performance) have not yet been found.

What are we going to do? In the middle, I tried another tool, extundelete, which is basically the same as ext3grep syntax, and the principle should be the same, but it is said that it can be restored by directory.

All right, give it a try:

Extundelete / dev/vgdata/LogVol00-- restore-directory var/lib/mysql/aqsh

As expected, I can't recover! Those documents have been destroyed. Report to the leader and implement Plan B. I had no choice but to go home from work. (it's the weekend. Go back and have a rest and think of something.)

Brainstorm: Binlog

I woke up early the next morning (something on my mind), carried my computer and went to the company (this weekend is reimbursed, free of criticism, notice, fine, dismissal, what weekend?).

Still running ext3grep,extundelete, just a few tricks, put the system on the test server and see if the data can find a way to fix it.

Mysqldump on the test server, restore the files, overwrite the recovered files, add permissions to the files, and restart MySQL.

Wait,Wait, isn't there a Binlog? All of our services are required to enable Binlog. Maybe we can recover data through Binlog?

So find the Binlog file in the file name from Dump, a total of three:

Mysql-binlog0001

Mysql-bin.000009

Mysql-bin.000010

Restore 0001:

Ext3grep / dev/vgdata/LogVol00-- restore-file var/lib/mysql/mysql-bin.000001

He failed unexpectedly. Look at the other two files, mysql-bin.000010 about a few hundred MB, should be more reliable, the implementation of the restore command, unexpectedly successful!

Quickly SCP to the test server. Perform a Binlog restore:

Mysqlbinlog / usr/mysql-bin.000010 | mysql- uroot-p

Enter the password, it is stuck (good sign), after a long wait, it is finally over. Open the app, oh, thanks to CCTV,MTV, the data is back!

Postscript

After the accident, although the data was lucky to be found, the process was thrilling. They are also afraid of the consequences of their own mistakes and the joint and several liability they bring to colleagues and leaders.

I also hope to remember this accident and not make the same mistake in the future. The reflection on the accident is as follows:

When arranging MM for server maintenance, I didn't explain the situation to her in advance, and I didn't pay attention to it either. The management was chaotic and the process was chaotic. In an online production system, any change must be planned first and then moved later.

There is a problem with the automatic backup and no one is checking it. Offline backup personnel never pay attention to downloading 1K files from the server every time. You need to be clear about your responsibilities at work.

After the accident, it was not found in time, resulting in part of the data written to disk, resulting in unrecoverable problems. You need to write an application monitoring program, and if there is an exception in the service, SMS will alert the relevant responsible person.

According to the comment reminder, add another note: you cannot use Root users to operate. Users with different permission levels should be set up on the server.

Through this accident, several colleagues who had nothing to do with the project and the accident offered to help, check the information, and help with the test, and one colleague even helped to conduct the data recovery test at more than 1 o'clock in the evening.

At the same time, when the product manager thinks of the great pressure on the customer, he does not panic and blame the developer and the specific operator, but allows everyone to calm down and think about the solution.

The department leaders also take the initiative to help think of ways to accompany us to work overtime tests and track the progress of things in real time. Through everyone's joint efforts, things finally came to a relatively successful end. next, we will conduct a collective reflection on Monday morning and sum up experiences and lessons, and we will try our best to avoid such accidents.

Links to the tools used in this article:

Ext3grep: https://code.google.com/p/ext3grep/

Compilation and installation depends on more packages, you can go to the Internet to search how to install. Unfortunately, the howto given by the author has been blocked. I FQ downloaded the pdf document of howto. After reading it, you will have a better understanding of Linux's file system.

This tool has a Bug that will not be executed down after an error:

Ext3grep: init_directories.cc:534: void init_directories (): Assertion `lost_plus_found_directory_iter! = all_directories.end () 'failed.

As a result, the recovery failed, and the author released a patch, download address: patch download. I don't understand why the author didn't include this patch in the new edition.

The function is similar to that of ext3grep, and the principle should be similar. Only claimed to be able to restore the directory, I did not succeed here.

At this point, I believe you have a deeper understanding of "how to solve the rm-rf problem in Oracle database". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.