Shulou(Shulou.com)06/01 Report--

Preface

The development of one of the business lines of a business group maintained some time ago found the operation and maintenance staff and raised a MongoDB optimization problem. at that time, MongoDB was being transferred from op management to db for maintenance, the whole department lacked experience in MongoDB operation and maintenance, and the optimization of MongoDB was an unknown challenge. When op found me, the public service platform of the core system, the MongoDB cluster used for SMS, wanted to make an optimization, and I couldn't let it take on a task that I thought I might not be able to handle.

The developer found that I raised two questions and asked the operation and maintenance team to solve the problem, but in the end, under the collision of my rational thinking and his emotional thinking, I finally won as a winner. I succeeded in persuading him and answered some of his questions. I got a satisfactory answer and never called me again. Of course, it certainly won't be based on a few words, no matter how rich your theory is, how ambiguous your attitude is, how you can convince experienced developers to do something without taking some real data. I communicated for most of the day and occupied my working time during the day, but the questions raised by him are still worth discussing.

According to the logic of the development, we want to expand the secondary node horizontally, put other less demanding services on the secondary node, reduce the pressure on the primary node, achieve partial read-write separation, and give priority to the main business. I think this is a good starting point, but I did not respond to it. One is that he did not realize that the delay is not the problem of database clustering (the next article will talk about the writing and business logic of MongoDB), and the second is that we really lack effective resource hardware to expand nodes.

Different business scenarios apply different architecture strategies. Sometimes, expanding secondary nodes can not solve the problem, especially those with high real-time performance, but sometimes expanding secondary nodes is really effective, such as service migration after hardware upgrade, and online expansion of secondary nodes is needed to meet the higher hardware requirements of the business.

To sum up, there are two ways to expand MongoDB's secondary node:

1. Direct extension of rs.add ()

2. Expand after consistent backup (personal name)

1. Rs.add ("HOST_NAME:PORT")

The specific implementation is to log in to the machine of the expansion node, edit the configuration file, establish the corresponding directory and permissions, and start the MongoDB instance.

It should be noted that this expansion method ensures the data level of the synchronization source, that is, the oplog of the MongoDB will not be overwritten before the data is synchronized, which is similar to the redo log log of MySQL. If the synchronized data is overwritten, the synchronization fails.

Another point to note is that in the process of synchronizing data, when the cluster data reaches a certain order of magnitude, the large size of the synchronized data will cause certain pressure on the network and may affect the core switch of the business, so it is necessary to use TC tools to limit the speed of synchronous traffic. This speed limit needs to be considered that the synchronization source may not be primary, or it may be the secondary node of the same role, so that the synchronization of external speed limit is bound to increase the synchronization time, which will increase the probability that the oplog will be covered. The specific speed limit can only be determined by calculation.

2. Quickly add secondary nodes to consistent snapshots (self-naming, welcome to communicate)

A) consistent snapshot backup on the primary node

B) perform consistent snapshot recovery on the secondary node, restoring only part of the data, not oplog for the time being

C) initialize the oplog.rs collection and restore oplog records

D) initialize the other two collections db.replset.election,db.system.replset of the local database

E) modify the database configuration and restart the database (before this step, the instance does not enable the authentication mode and the configuration of the replication set), rs.add ("HOST_NAME:PORT") adds secondary to the cluster and observes the synchronization status and verifies the integrity and consistency of the data

The detailed practice process of the practice is as follows (for reference and communication only, the production environment should be used with caution):

1. Perform consistent snapshot backup on primary

# primary node or other secondary node backup data [root@172-16-3-190 mongodb] # / opt/app/mongodb/bin/mongodump-uroot-ppwd4mysql-- authenticationDatabase=admin-- port=27017-- oplog-o / tmp/dump_mongo/2018-08-20T15:42:47.028+0800 writing admin.system.users to 2018-08-20T15:42:47.030+0800 done dumping admin.system.users (1 document) 2018-08-20T15:42:47.030+0800 writing admin.system.version to 2018 -08-20T15:42:47.031+0800 done dumping admin.system.version (2 documents) 2018-08-20T15:42:47.032+0800 writing super_hero.user_address to 2018-08-20T15:42:47.032+0800 writing super_hero.user_info to 2018-08-20T15:42:47.033+0800 done dumping super_hero.user_address (1 document) 2018-08-20T15:42:47.033+0800 done dumping super_hero.user_info (1 document) 2018-08-20T15:42:47.034+0800 Writing captured oplog to 2018-08-20T15:42:47.036+0800 dumped 1 oplog entry# View backed up files [root@172-16-3-190 mongodb] # ls-lh / tmp/dump_mongo/total 12Kdrwxr-xr-x 2 root root 4.0K Aug 20 15:42 admin-rw-r--r-- 1 root root 110 Aug 20 15:42 oplog.bsondrwxr-xr-x 2 root root 4.0K Aug 20 15:42 super_hero# pass the backup to the one you are going to add as secondary [root@172-16-3-190 tmp] # scp-r-P22222 / tmp/dump_mongo/ liyingxiao@172.16.3.189:/tmp on the node

2. Secondary node consistent snapshot restore

# auth=true#replSet = repl_mongo#clusterAuthMode=keyFile#keyFile=/opt/app/mongodb/keyfile/mongodb.key## recovery data [root@172-16-3-189 we_ops_admin] # / opt/app/mongodb/bin/mongorestore-- oplogReplay-- port=27017 / tmp/dump_mongo/ 2018-08-20T15:56:32.161+0800 preparing collections to restore from2018-08-20T15:56:32.193+0800 reading metadata for super_hero.user_info from / tmp/dump_mongo/super_ Hero/user_info.metadata.json2018-08-20T15:56:32.194+0800 reading metadata for super_hero.user_address from / tmp/dump_mongo/super_hero/user_address.metadata.json2018-08-20T15:56:32.222+0800 restoring super_hero.user_address from / tmp/dump_mongo/super_hero/user_address.bson2018-08-20T15:56:32.300+0800 restoring super_hero.user_info from / tmp/dump_mongo/super_hero/user_info.bson2018-08 -20T15:56:32.867+0800 no indexes to restore2018-08-20T15:56:32.867+0800 finished restoring super_hero.user_address (1 document) 2018-08-20T15:56:32.881+0800 no indexes to restore2018-08-20T15:56:32.881+0800 finished restoring super_hero.user_info (1 document) 2018-08-20T15:56:32.881+0800 restoring users from / tmp/dump_mongo/admin/system.users.bson2018-08-20T15:56:32.993+0800 replaying oplog2018-08-20T15:56:32.997+0800 done

3. Initialize the oplog.rs collection and restore the oplog record

Create oplog.rs set merge initialization size

Use local db.createCollection ("oplog.rs", {"capped": true, "size": 100000000})

Restore data from the oplog.rs collection of consistent backups to the secondary node

[root@172-16-3-189 we_ops_admin] # / opt/app/mongodb/bin/mongorestore-d local-c oplog.rs-- port=27017 / tmp/dump_mongo/oplog.bson2018-08-20T16:12:49.848+0800 checking for collection data in/ tmp/dump_mongo/oplog.bson2018-08-20T16:12:49.852+0800 restoring local.oplog.rs from / tmp/dump_mongo/oplog.bson2018-08-20T16:12:49.925+0800 no indexes to restore2018-08-20T16 : 12Plus 49.9250800 finished restoring local.oplog.rs (1 document) 2018-08-20T16:12:49.925+0800 done

4. Initialize the db.replset.election,db.system.replset collection, where the replset.election needs to query the master node data and store the data in the secondary node, or combine the two to save to the secondary node. Another set system.replset can automatically identify the content of the primary node after joining the replication set (here I synchronize the data by myself)

# primary node repl_mongo:PRIMARY > db.replset.election.find () {"_ id": ObjectId ("5b7a6ee5de7a24b82a686139"), "term": NumberLong (1), "candidateIndex": NumberLong (0)} # secondary node db.replset.election.save ({"_ id": ObjectId ("5b7a6ee5de7a24b82a686139"), "term": NumberLong (1), "candidateIndex": NumberLong (0)})

5. Modify the database configuration and restart, and add the secondary node to the replication cluster

# auth=true#replSet = repl_mongo#clusterAuthMode=keyFile#keyFile=/opt/app/mongodb/keyfile/mongodb.key [root@172-16-3-189 we_ops_admin] # / opt/app/mongodb/bin/mongod-- shutdown-f / opt/app/mongodb/mongo.conf killing process with pid: 5331 [root@172-16-3-189 we_ops_admin] # vim / opt/app/mongodb/mongo.conf # comment removed and restart [root@172-16-3-189 we _ ops_admin] # / opt/app/mongodb/bin/mongod-f / opt/app/mongodb/mongo.conf about to fork child process Waiting until server is ready for connections.forked process: 5722child process started successfully, parent exiting# add the secondary node repl_mongo:PRIMARY > rs.add ({"_ id": 1, "host": "172.16.3.189 Timestamp 27017"}) {"ok": 1, "operationTime": Timestamp (1534752953, 1), "$clusterTime": {"clusterTime": Timestamp (1534752953, 1), "signature": {"hash": BinData (0, "Tt9nzhoVYdUtGFZnc1Kg1exl0Hc=") "keyId": NumberLong ("6591702943026642945")}}

6. Log in to the added secondary node to verify the replication set status, data integrity and consistency.

[root@172-16-3-189 we_ops_admin] # / opt/app/mongodb/bin/mongo-uroot-ppwd4mysql-- authenticationDatabase=admin-- port=27017

This paper focuses on the second method of adding secondary nodes, which saves time and effort. In practice, the database instance removes authentication and replication set parameters in the early stage, which is convenient for some operations that require user permissions to avoid establishing administrator accounts. After joining the cluster, the accounts of primary nodes are synchronized by themselves. When logging in to the secondary node after restart to verify the availability and data consistency of the service, use the cluster's management account to enter, otherwise an authentication error will be reported.

To sum up the above two expansion methods, the expansion of mode 1 is simple and easy, and we need to ensure that oplog is not covered and evaluate the impact of synchronous traffic, which is our usual method of adding secondary nodes to horizontal replication sets. For the second method, the operation is tedious but do not worry about oplog coverage, and do not worry too much about network traffic during the operation, only consider the impact of network traffic. The first method has a long operation time period, a large range of uncontrollable influence, time-consuming and energy-consuming, and the second mode has a short operation time and many steps of operation, which is prone to other problems.

The MongoDB secondary node has a recovering state

After MongoDB has done replica sets, the secondary node appears recovering state

After a mongo cluster hangs and restarts, it is found that the mongo node of one server has been in the recovering state and cannot become secondary or primary.

After consulting the official documentation, find the solution and record it here.

Cause of occurrence

The working principle of the backup node can be roughly described as: the backup node regularly polls the data operations on the primary node, and then carries out these operations on its own data copies, so as to ensure the data synchronization with the primary node.

As for all database state changes on the primary node, they are stored in a specific system table. The backup node updates its own data based on these data.

The operation of the database state change mentioned above is called oplog (operation log, master node operation record). Oplog is stored in the "oplog.rs" table in the local database. The backup node in the replica set asynchronously synchronizes the oplog from the master node, and then re-performs the operations recorded by it, so as to achieve the role of data synchronization.

There are a few things to note about oplog:

Oplog only records operations that change the state of the database. The operations stored in oplog are not exactly the same as those performed by the master node. For example, the "$inc" operation is transformed into a "$set" operation. Oplog is stored in a fixed set (capped collection). When the number of oplog exceeds oplogSize, the new operation will overwrite the old operation.

Data synchronization

In a replica set, there are two ways to synchronize data:

Initial sync (initialization): this process occurs when a new database is created in the replica set or one of the nodes has just recovered from the downtime, or when a new member is added to the replica set, by default, the node in the replica set replicates oplog from the nearest node to synchronize data, which can be primary or the secondary node with the latest oplog replica. This operation generally reinitializes the backup node, which is expensive replication (replication): this operation continues after initialization to keep the data synchronized between the secondary nodes.

Initial sync

When you encounter a problem that cannot be synchronized in the above example, you can only use the following two ways to initial sync

The first way is to stop the node, then delete the files in the directory, and restart the node. In this way, the node will execute the initial sync

Note: in this way, the sync time is based on the amount of data. If the amount of data is too large, the sync time will be very long.

At the same time, there will be a lot of network transfers, which may affect the work of other nodes. Second, stop the node, then delete the files in the directory, find a relatively new node, and then copy the files in the node directory to the node directory that you want to sync.

Summary

The above is the whole content of this article, I hope that the content of this article has a certain reference and learning value for your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.