Implementation of mongodb incremental backup script

Implementation of mongodb incremental backup script? In view of this problem, this article introduces the corresponding analysis and answers in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible way.

In the mongodb cluster architecture, the slave library is synchronized with the master library by asynchronously copying the Oplog files of the master library.

Oplog records the change operation information of the MongoDB database. It is stored in the oplog.rs table of the local database and only exists in the cluster architecture. Therefore, incremental backup cannot be used on a stand-alone machine. Oplog has a size limit, exceeding the specified size, and the new record will overwrite the old operation record.

Principle of mongodb incremental backup

How to download oplog for a certain period of time, I stitched together an example:

Mongodump-h 127.0.0.1-- port 27117-d local-c oplog.rs-u admin-p xxx-- authenticationDatabase admin-Q'{ts: {$gt: {$timestamp: {timestamp 1451355000 timestamp 1}}, $lt: {timestamp: {timestamp 1451357430 timestamp 1}}, ns:/ ^ test _ db\. /}'- o oplog_backup

The above is the oplog exported between 1451355000 and 1451357430. To export oplog, you need to switch to an account with admin permission.

The principle is very simple, but the specific implementation still requires a lot of consideration, specific look at the code. The script backs up the full database once in a cycle (such as a week), followed by an incremental backup each time. Script address: http://git.oschina.net/passer/mongodb_backup_script

The process when the incremental script is executed

Read the execution information of the previous cycle to determine whether a new cycle needs to be created

Get the point-in-time current timestamp position recently recorded by oplog on mongodb

Read the oplog time point of the last execution mongodb locally

Dump exports all data or incremental oplog files locally. The export range of incremental oplog files is the oplog file from the last oplog record point to the latest time.

Save the current timestamp position obtained in step 2 locally as the point in time for the next execution of step 3

Compress

Upload to oss

Delete local backup files

The process when a full script is executed

Back up the mongodb database locally

Compress

Upload to oss

Verify that the size of the oss is the same as the local file

Delete local backup files

The process that the script executes during recovery

Download backup files for the specified cycle from oss to local

Decompress the zip files of full files and incremental oplog

Import all files with mongorestore

Import the oplog files of each time period with mongorestore-- oplogReplay

This is the answer to the question about the implementation of the mongodb incremental backup script. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel to learn more about it.