Shulou(Shulou.com)06/01 Report--

Problem encountered: an alarm was received in the early morning, and the cpu load of a mongodb server exceeded 8. As it did not affect the business, we began to investigate the cause the next morning.

Check the cause:

1. Learn about the applications on the server first.

The main application of the db server is mongodb. So start querying the mongodb log:

Through the early morning log mongodb.log view, there are a large number of slow queries, but in fact these tables are very small, only a few hundred rows of data, and the table has an index, but only one query took 60 seconds 80s. The log after the slow query shows that the node is not detected by other nodes (in the form of mongodb replication set).

Since queries in a small table need to be judged for about 70 seconds, and mongodb deploys replication sets (other query nodes are normal), it may be the performance of this db that affects the mongodb rather than the mongodb itself.

two。 So check to see if there are any tasks before running in the early hours of the morning.

Crontab-l found that there was indeed a mission in the early hours of the morning. It's a script for cutting logs. Probably just cp the log to another directory, then empty the current log and continue to log for the new day.

But this log is usually small and has been running for a long time. You can only try. Look at the log directory.

I saw that the log was suddenly so big. Is it because of the cp file at night?

It is almost concluded that the problem is that cp causes the io load to be too high, which in turn leads to too high cpu load.

3. Recurrence problem

Because the operation is operated at night, the online business is not affected. Therefore, manual operation of cp, reproduce the problem.

Cp has a file of nearly 3G, as shown in the following figure: you can see that too many requests have been generated, the system is fully loaded, and there may be a bottleneck on the disk.

Then cpu load rises to around 10 (cpu load value per minute).

Resolve the problem:

Separate the larger logs from the mongodb server.

Change the small daily log script cutting to the system logrotate cutting. Logrotate operates when the system is idle.

But why copy a 3G file, it will lead to cpu load as high as 10. 5%. This leads to a sharp decline in the performance of mongodb queries.

So we contacted a cloud, how can the performance of an ordinary cloud disk be so low!

The above ideas for investigating problems did not go very well at the beginning. At first, the file was small and did not guess that it was a log copy. When looking up problems, you can't rule them out with habitual thinking.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.