Shulou(Shulou.com)06/01 Report--

The background project stores data in the form of mongo replica set, which often leads to the problem that a copy mongos cannot be started due to an unexplained power outage. Environment introduction

Mongodb copy set:

Mongo01:192.168.36.218

Mongo02:192.168.36.219

Mongo03:192.168.36.220

Error message

The mongos on the mongo03 copy cannot be started. An error was reported when executing the command:

[root@localhost] # mongos-- configdb 192.168.36.218 configdb 20000192.168.36.219 Vera 20000192.168.36.220 port 30000-- chunkSize 30000-- logpath / home/mongo/logs/mongos.log-- logappend-- forkabout to fork child process, waiting until server is ready for connections.forked process: 79748ERROR: child process failed, exited with error number 5

Check the mongos.log and the error message is as follows:

2018-06-25T09:13:47.607+0800 I CONTROL [main] * SERVER RESTARTED * 2018-06-25T09:13:47.612+0800 I CONTROL [main] * * WARNING: You are running this process as the root user Which is not recommended.2018-06-25T09:13:47.613+0800 I CONTROL [main] 2018-06-25T09:13:47.613+0800 I SHARDING [mongosMain] MongoS version 3.2.1 starting: pid=80904 port=30000 64-bit host=MongoDB03 (--help for usage) 2018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] db version v3.2.12018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] git version: a14d55980c2cdc565d4704a7e3ad37e4e535c1b22018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] Allocator: tcmalloc2018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] modules: none2018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] build environment:2018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] distarch: x86 * 642018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] target_arch: x86 * * 642018-06-25T09:13:47.613+0800 I CONTROL [mongosMain] options: {net: {port: 30000} ProcessManagement: {fork: true}, sharding: {chunkSize: 500, configDB: "192.168.36.218 sharding 20000192.168.36.219VERV 20000192.168.36.220lv 20000192.168.36.220"}, systemLog: {destination: "file", logAppend: true Path: "/ home/mongo/logs/mongos.log"} 2018-06-25T09:13:47.613+0800 I SHARDING [mongosMain] Updating config server connection string to: 192.168.36.218VV 20000192.168.36.219VOR 20000192.168.36.220RV 200002018-06-25T09:13:47.625+0800 W SHARDING [mongosMain] config servers 192.168.36.218VOV 20000 and 192.168.36.220V differ2018-06-25T09:13:47.627+0800 W SHARDING [mongosMain] config Servers 192.168.36.21825T09:13:47.628+0800 and 192.168.36.22000 differ2018-06-25T09:13:47.628+0800 W SHARDING [mongosMain] config servers 192.168.36.218SHARDING 20000 and 192.168.36.22025T09:13:47.630+0800 W SHARDING [mongosMain] config servers 192.168.36.218mongosMain and 192.168.36.22000 differ2018-06-25T09:13:47.630+0800 E SHARDING [mongosMain] Error initializing sharding system: ConfigServersInconsistent hash from 192. 168.36.218 d41d8cd98f00b204e9800998ecf8427e 20000: {chunks: "d41d8cd98f00b204e9800998ecf8427e" Databases: "95954cb16c029767f4ad050712a28f49", shards: "68f4b37fec8c2ac97cc985aa01f37717", version: "b25e55c19a8c75c87b4f950dcf5eb088"} vs hash from 192.168.36.220 68f4b37fec8c2ac97cc985aa01f37717 20000: {}

As mentioned above, we find that:

Config servers 192.168.36.218:20000 and 192.168.36.220:20000 differ

The configuration server 192.168.36.218 20000 is inconsistent with the damaged configuration server 192.168.36.220 20000, which means that the configuration above mongo01:20000 and mongo03:20000 is not the same.

Repair

So how do we fix it? In fact, we can import the config library on mongo01:20000 into the config library of mongo03:20000 to solve the above problem.

Execution process

Step 1: back up mongo01:20000 's config library

[root@localhost] # mongodump-- host 192.168.36.218purl 20000-d config-o / home/config2018-06-25T10:06:06.225+0800 writing config.actionlog to 2018-06-25T10:06:06.226+0800 writing config.locks to 2018-06-25T10:06:06.226+0800 writing config.mongos to 2018-06-25T10:06:06.226+0800 writing config.lockpings to 2018-06-25T10:06:06.227+0800 done dumping config.locks (3 documents) 2018 06-25T10:06:06.228+0800 done dumping config.lockpings (2 documents) 2018-06-25T10:06:06.229+0800 done dumping config.mongos (3 documents) 2018-06-25T10:06:06.229+0800 writing config.shards to 2018-06-25T10:06:06.229+0800 writing config.settings to 2018-06-25T10:06:06.229+0800 writing config.version to 2018-06-25T10:06:06.230+0800 done dumping config.shards (1 document) 2018-06-25T10:06: 06.2300800 writing config.databases to 2018-06-25T10:06:06.230+0800 done dumping config.settings (1 document) 2018-06-25T10:06:06.230+0800 done dumping config.version (1 document) 2018-06-25T10:06:06.230+0800 writing config.changelog to 2018-06-25T10:06:06.230+0800 writing config.chunks to 2018-06-25T10:06:06.231+0800 done dumping config.databases (1 document) 2018-06-25T10:06:06.231+0800 writing config .tags to 2018-06-25T10:06:06.232+0800 done dumping config.chunks (0 documents) 2018-06-25T10:06:06.232+0800 done dumping config.changelog (1 document) 2018-06-25T10:06:06.232+0800 done dumping config.tags (0 documents) 2018-06-25T10:06:06.355+0800 done dumping config.actionlog (8160 documents) [root@localhost] #

Step 2: import the backed-up config into mongo03:20000

[root@localhost] # mongorestore-- host 192.168.36.220 host 20000-d config/ home/config/config2018-06-25T10:08:09.136+0800 building a list of collections to restore from / home/config/config dir2018-06-25T10:08:09.162+0800 reading metadata for config.actionlog from / home/config/config/actionlog.metadata.json2018-06-25T10:08:09.163+0800 reading metadata for config.locks from / home/config/config/locks.metadata.json2018-06-25T10 : 08VR 09.16350800 reading metadata for config.changelog from / home/config/config/changelog.metadata.json2018-06-25T10:08:09.164+0800 reading metadata for config.mongos from / home/config/config/mongos.metadata.json2018-06-25T10:08:09.164+0800 restoring config.locks from / home/config/config/locks.bson2018-06-25T10:08:09.165+0800 restoring config.mongos from / home/config/config/mongos.bson2018-06-25T10:08:09.175+0800 error : multiple errors in bulk operation:-E11000 duplicate key error collection: config.mongos index: _ id_ dup key: {: "MongoDB01:30000"}-E11000 duplicate key error collection: config.mongos index: _ id_ dup key: {: "MongoDB02:30000"} 2018-06-25T10:08:09.175+0800 restoring indexes for collection config.mongos from metadata2018-06-25T10:08:09.204+0800 finished restoring config.mongos (3 documents) 2018-06-25T10:08: 09.204 0800 restoring indexes for collection config.locks from metadata2018-06-25T10:08:09.204+0800 reading metadata for config.lockpings from / home/config/config/lockpings.metadata.json2018-06-25T10:08:09.205+0800 restoring config.lockpings from / home/config/config/lockpings.bson2018-06-25T10:08:09.215+0800 restoring config.actionlog from / home/config/config/actionlog.bson2018-06-25T10:08:09.269+0800 restoring config.changelog from / home/config/config/changelog. Bson2018-06-25T10:08:09.284+0800 restoring indexes for collection config.changelog from metadata2018-06-25T10:08:09.284+0800 finished restoring config.locks (3 documents) 2018-06-25T10:08:09.284+0800 reading metadata for config.shards from / home/config/config/shards.metadata.json2018-06-25T10:08:09.284+0800 restoring config.shards from / home/config/config/shards.bson2018-06-25T10:08:09.286+0800 finished restoring config.changelog (1 document) 2018- 06-25T10:08:09.286+0800 reading metadata for config.version from / home/config/config/version.metadata.json2018-06-25T10:08:09.286+0800 restoring config.version from / home/config/config/version.bson2018-06-25T10:08:09.287+0800 error: multiple errors in bulk operation:-E11000 duplicate key error collection: config.lockpings index: _ id_ dup key: {: "MongoDB01:30000:1523346496:1804289383"}-E11000 duplicate key error collection: config. Lockpings index: _ id_ dup key: {: "MongoDB02:30000:1529566939:1804289383"} 2018-06-25T10:08:09.287+0800 restoring indexes for collection config.lockpings from metadata2018-06-25T10:08:09.297+0800 restoring indexes for collection config.shards from metadata2018-06-25T10:08:09.318+0800 finished restoring config.lockpings (2 documents) 2018-06-25T10:08:09.318+0800 reading metadata for config.databases from / home/config/config/databases.metadata.json2018-06 -25T10:08:09.318+0800 restoring config.databases from / home/config/config/databases.bson2018-06-25T10:08:09.330+0800 restoring indexes for collection config.version from metadata2018-06-25T10:08:09.370+0800 finished restoring config.shards (1 document) 2018-06-25T10:08:09.370+0800 reading metadata for config.settings from / home/config/config/settings.metadata.json2018-06-25T10:08:09.370+0800 restoring config.settings from / home/config/config/settings. Bson2018-06-25T10:08:09.372+0800 finished restoring config.version (1 document) 2018-06-25T10:08:09.372+0800 reading metadata for config.tags from / home/config/config/tags.metadata.json2018-06-25T10:08:09.372+0800 restoring config.tags from / home/config/config/tags.bson2018-06-25T10:08:09.374+0800 restoring indexes for collection config.tags from metadata2018-06-25T10:08:09.382+0800 restoring indexes for collection config.databases from metadata2018-06 -25T10:08:09.400+0800 finished restoring config.tags (0 documents) 2018-06-25T10:08:09.400+0800 reading metadata for config.chunks from / home/config/config/chunks.metadata.json2018-06-25T10:08:09.400+0800 restoring config.chunks from / home/config/config/chunks.bson2018-06-25T10:08:09.409+0800 restoring indexes for collection config.settings from metadata2018-06-25T10:08:09.409+0800 finished restoring config.databases (1 document) 2018-06-25T10 : 0825T10:08:09.555+0800 restoring indexes for collection config.chunks from metadata2018 09.4110800 finished restoring config.settings (1 document) 2018-06-25T10:08:09.555+0800 restoring indexes for collection config.chunks from metadata2018-06-25T10:08:09.581+0800 finished restoring config.chunks (0 documents) 2018-06-25T10:08:09.754+0800 restoring indexes for collection config.actionlog from metadata2018-06-25T10:08:09.754+0800 finished restoring config.actionlog (8160 documents) 2018-06-25T10:08:09.754+0800 done [root@localhost ~] #

Note: I executed in a production environment, backing up all the config libraries on mongo01:20000 and mongo02:20000, and then importing them into mongo03:20000 sequentially (backup and import commands are the same as above). In theory, it should only import mongo01:20000, because it is a production environment, I will not test it, wait until the next time this problem occurs and try again. At the same time, you are welcome to test it, and then reply to the comments section to help you solve the problem.

Try to launch mongos [root@localhost ~] # mongos on mongo03-- configdb 192.168.36.218 mongos 20000192.168.36.219 mongos 20000192.168.36.220 mongos-- port 30000-- chunkSize 30000-- logpath / home/mongo/logs/mongos.log-- logappend-- forkabout to fork child process, waiting until server is ready for connections.forked process: 4286child process started successfully Parent exiting [root@localhost ~] # ps-ef | grep mongosroot 44760 10 Jun21? 00:25:14 mongos-- configdb 192.168.36.218VV 20000192.168.36.219VOO 20000192.168.36.220RV 20000-- port 30000-- chunkSize 30000-- logpath / home/mongo/logs/mongos.log-- logappend-- forkroot 66128 660900 11:40 pts/0 00:00:00 grep mongos [root@localhost] #

As above, we can see that mongos starts successfully and keeps running in the background.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.