Shulou(Shulou.com)05/31 Report--

This article mainly introduces mongodb how to achieve incremental / full backup script, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

Function

Perform full or incremental backups of mongodb database data on a regular basis (replica set architecture), and can be compressed and uploaded to Aliyun oss (local will become a compressed package, you can set not to upload oss).

Script running environment

Written in python, python,pymongo and mongodb shell clients need to be installed (python 2.7.6 and mongodb shell 2.0.4 are used for testing).

Script deployment steps

1. Put the script on a linux host

two。 In the case of an incremental backup, create a mongodb backup role user or a higher privileged admin library user. (when exporting, you will first switch to the admin library to verify permissions. You need the permission to query the local library, mongodump local library and mongodump target library)

Use admin db.addUser ({user: "xxxxx", pwd: "xxxxx", roles: ["backup"]})

3. Edit config.properties, modify oss, mongodb connection and other configuration information

# # Aliyun oss connection configuration endpoint= oss.aliyuncs.com accessKeyId = xxxxxxx accessKeySecret = xxxxxxx bucket = db-backup # # mongodb connection configuration # it is recommended to use the address of the slave library to reduce the pressure on the master database db_host= localhost db_port= 27017 # if it is an incremental backup scheme, the user created in step 2, or the admin user with higher privileges If it is a full backup scheme, you only need to have the operation rights of the target library db_user= testb db_passwd= testb # target library db_name= che # backup to the local temporary directory db_backup_root_path= / temp/backup/ # if you use the green version of the mongo client Write the absolute path of the mongo client: mongo_shell_path= / dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/ # incremental backup or full backup 1: incremental backup 0: full backup is_inc_backup=1 # how many days does the full backup full_backup_period=7 # upload to oss, if 1, the local backup file will be deleted after the upload is successful 0: do not upload to oss is_upload_to_oss= 0

4. Add start.py to the linux scheduled task. Crontab task configuration such as 0 4 * python / xxx/start.py > > / xxx/xxx.log 2 > & 1

Recovery steps in increment:

1. Create a role with applyOps permissions for mongodb and the users who use this role. (user permission to execute mongorestore-oplogReplay is required)

Use admin db.createRole ({role: "applyOpsRole", privileges: [{resource: {anyResource: true}, actions: ["anyAction"]}], roles: []}) db.addUser ({user: "xxxx", pwd: "xxxx", roles: ["applyOpsRole"]})

two。 Modify the configuration in restore_inc.py

# # Aliyun oss configuration endpoint= "oss.aliyuncs.com" accessKeyId= "xxxxxxx" accessKeySecret= "xxxxxxx" bucket= "db-backup" # # mongodb imported configuration db_host= "localhost" db_port=27017 # user db_user= "testr" db_passwd= "testr" db_name= "che" # recent circle backup direactory on oss's latest backup file cycle name That is, to back up the last_circle_backup_dir_name of mongodb_inc_backup_info.json in the temporary directory or the folder name of last_circle_backup_dir_name= "mongodb_cycle_backup_20151124141133" # download from oss to the local temporary directory restore_local_temp_path= "H:\\ pythoncode\\ temp\\ restore\" # if you use the green version of the mongo client Write the absolute path of the mongo client mongo_shell_path= "/ alidata1/dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/" # backup file has download to local? Whether the backup file of if True,will not download backup files from oss # has been downloaded locally, if true, it will not be downloaded and unzipped from oss, and whether the old database is_drop_old_restore=True will be deleted first when has_download_to_local=False # is restored.

3. Stop mongodb writing during import

4. Execute restore_inc.py

Full recovery steps:

1. Modify the configuration in restore_full.py

# # Ali Cloud oss configuration endpoint= "oss.aliyuncs.com" accessKeyId= "xxxxxxx" accessKeySecret= "xxxxxxx" bucket= "db-backup" # # mongodb imported configuration db_host= "localhost" db_port=27017 # database corresponding to the user db_user= "test" db_passwd= "test" db_name= "che" # recent circle backup direactory on oss cycle name of the latest backup file The name of the file stored on oss is the directory where last_circle_backup_dir_name+last_full_backup_file_suffix last_circle_backup_dir_name= "mongodb_cycle_backup_20151124141133" last_full_backup_file_suffix= ".tar.gz" # was backed up The path of the actual full backup is restore_local_temp_path+last_circle_backup_dir_name+db_name restore_local_temp_path= "H:\\ pythoncode\\ temp\\ restore\\" # if you use the green version of the mongo client, write the absolute path of the mongo client mongo_shell_path= "/ alidata1/dev/hanxuetong/mongodb/mongodb-linux-x86_64-3.0.6/bin/" # backup file has download to local? Whether the backup file of if True,will not download backup files from oss # has been downloaded locally, if true, it will not be downloaded and unzipped from oss, and whether the old database is_drop_old_restore=True will be deleted first when has_download_to_local=False # is restored.

two。 Execute restore_full.py

Related:

Implementation principle of incremental backup

Back up the full database once in a cycle (such as a week), and then back up the oplog files from the last record point to the latest time each time. Oplog records the change operation information of the MongoDB database. It is stored in the oplog.rs table of the local database and only exists in the cluster architecture. Therefore, incremental backup cannot be used on a stand-alone machine. The slave library is synchronized with the master library by asynchronously copying the Oplog files of the master library. Oplog has a size limit, exceeding the specified size, and the new record will overwrite the old operation record.

The process when a full script is executed

Back up the mongodb database locally

Compress

Upload to oss

Verify that the size of the oss is the same as the local file

Delete local backup files

The process when the incremental script is executed

Read the execution information of the previous cycle to determine whether a new cycle needs to be created

Get the point-in-time current timestamp position recently recorded by oplog on mongodb

Read the oplog time point of the last execution mongodb locally

Dump exports all data or incremental oplog files locally. The export range of incremental oplog files is the oplog file from the last oplog record point to the latest time.

Save the current timestamp position obtained in step 2 locally as the point in time for the next execution of step 3

Compress

Upload to oss

Delete local backup files

The process that the script executes during recovery

Download backup files for the specified cycle from oss to local

Decompress the zip files of full files and incremental oplog

Import all files with mongorestore

Import the oplog files of each time period with mongorestore-- oplogReplay

Mongodb incremental backup script source code address:

Https://gitee.com/passer/mongodb_backup_script

Thank you for reading this article carefully. I hope the article "how to achieve incremental / full backup script for mongodb" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support and follow the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.