Shulou(Shulou.com)06/03 Report--

Last week, I noticed that a NLP server spent a lot of time completing the analysis request, and one of the CPU cores had a long-term occupancy rate of 100%. The response speed of each service is much slower than when it was first launched.

Therefore, the top of the system and the third-party htop are used to screen respectively, and finally locate the problem of the log segmentation service of the logrotate system.

The following is a record of the specific handling methods:

First, execute the following command:

$top

PID USER PR NI VIRT RES SHR S CPU MEM TIME+ COMMAND

6479 root 20 0 413m 409m 860 R 99 40.9 927:36.64

/ usr/sbin/logrotate / etc/logrotate.conf

$tail / var/lib/logrotate/logrotate.status

/ / learn from the status file that there are a large number of log processing tasks for tomcat. Tomcat does not configure custom log processing logic, and logs are hosted by the system. Moreover, the business processing of tomcat is very large, so there are too many tasks to be handled when using logrotate, resulting in CPU and disk being very busy.

$ls / var/lib/logrotate/logrotate.status-lh

-rw-r--r-- 1 root root 66m Aug 24 08:18 / var/lib/logrotate/status

The above command to get the entire status file unexpectedly as high as 156m, terrible!

Then, let's deal with the problem.

1) $ps aux | grep logrotate / / find out the process number of logr

2) $kill-9 process number / / stop logr

3) $/ usr/local/tomcat/bin/shutdown.sh / / stop tomcat

4) $rm-rf / usr/local/tomcat/logs / / Clean the log file under tomcat [pid may also be placed here, so let's stop tomcat first]

5) $rm-f / var/lib/logrotate/logrotate.status / / deleting the status log file of logr which is too large will also cause it to use 100%CPU

6) $logrotate-f / etc/logrotate.conf / / re-establish the status log file logrotate.status

7) restart tomcat / / everything is fine

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.