Shulou(Shulou.com)06/01 Report--

Recently, a MongoDB abnormal downtime failure was handled, related to the MongoDB engine parameter cacheSizeGB, which is used to limit the amount of memory used by MongoDB's wiredTiger engine. The following is the troubleshooting process, which can be used as a reference for future problem handling.

Feedback from the business side: MongoDB went down abnormally at 9:16:00 on September 16, 2019.9, and restarted normally by hand, but no exception was found when viewing the mongodb log.

Report an error. Https://wenku.baidu.com/view/4bac53c8326c1eb91a37f111f18583d048640f38

Problem troubleshooting process:

1. Observe the log of mongodb and find that except for the manual restart, there is no error in the log.

2. According to the running mechanism of mongodb, it consumes a lot of memory, and the application software running on the operating system goes down without error.

It is generally related to operating system resource failures, such as host memory, IO, network, and so on. Zhengzhou Infertility Hospital: http://yyk.39.net/zz3/zonghe/1d427.html

3. Check the host operating system log and find that there is an OOM alarm.

-- 9:17:10 on September 16, 2019.The operating system log prompt: mongod invoked oom-killer

-- 9:17:12 on September 16, 2019.The operating system log indicates that the operating system is running out of swap space.

At 9:17:13 on September 16, 2019.16, the operating system log indicates that 28232 of the mongod process was dropped by kill and occupied about 31 gigabytes of memory by kill.

4. Look at the operating system memory worker 32G

5. Check the startup configuration file of mongod. There is no parameter cacheSizeGB that limits the memory usage of mongod.

6. Analysis and summary:

2019.9.16 9, 9, 17, 10, 9, 17, 13, the host's memory and swap are exhausted one after another, causing oom-killer kill mongod and mongodb database.

He died without a disease. It is recommended that customers add the memory usage limit parameter cacheSizeGB of mongod to the startup configuration file of mongodb.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.