Shulou(Shulou.com)05/31 Report--

This article mainly analyzes the relevant knowledge points of the example analysis of the sys toolbox, the content is detailed and easy to understand, the operation details are reasonable, and has a certain reference value. If you are interested, you might as well follow the editor to have a look, and follow the editor to learn more about the sample analysis of the sys toolbox.

Introduction

Online operating environments sometimes encounter cpu surge scenarios. Generally speaking, for multi-core virtual machines, a common burst scenario is caused by high concurrency. When multi-core concurrency is high, syscall will consume high sys in locking. Of course, you can't do it only by guessing. Here are several common ghost hunting tools.

Toolbox

1. Nmon promes analysis

Promes, in particular, is recommended to provide three-dimensional system-level monitoring.

2. Perf analysis

Perf top-a-Gperf top-a-e cs-Gperf record-g-p 14778-e cycles sleep 10TIPS:perf sampling interface press shift + e expandable stack shift + c foldable state screenshot right arrow key to check its context (such as which lib it belongs to), left arrow key returns the best functions to drill down the screenshot

3. System call statistics

{top-H-p PID-b-n 1 | grep execname | awk'{print $1}'| sed's /\ ([0-9] *\) /-p\ 1 xargs strace g'| xargs strace-c 2 > / tmp/stat &}; sleep 5 and pkill strace

4. Call details

{top-H-p PID-b-n 1 | grep execname | awk'{print $1}'| sed's /\ ([0-9] *\) /-p\ 1 xargs strace g'| xargs strace-I-v-T 2 > / tmp/detail &}; sleep 5 against pkill strace

5. Sar-wq 1 10

6 、 vmstat 1 10

7. Mpstat-I ALL 1

8. Rpm-qa > / tmp/rpms

9. Stack dump

Echo 1 > / proc/sys/kernel/sysrqecho t > / proc/sysrq-trigger A case

When a database system imports data, the syscpu will be very high.

You can see that every time the task runs, sys will be high. Preliminary analysis shows that the target machine has 16c. According to experience, it is suspected that time is spent on parallel lock processing, but which part needs to use perf to grab the stack online.

You can see that sys time is consumed in spinlock, which is in the memory compaction and finishing routine of the MM system, and hugepage is opened in combination with this database, so it can be preliminarily concluded that it is because the memory is too fragmented to meet the allocation of giant pages, that is, there is not enough memory for high-level pages as a whole.

Of course, there is more than this phenomenon, and it is also noted that there are many processes created in the system at the moment of the problem, as high as hundreds per second, as shown in the following figure.

This does not match the 10% concurrency that the project team believes, so the comprehensive conclusion is that

Sys high time is mainly spent on memory defragmentation, which is mainly due to lack of memory. One of the accompanying phenomena is that threads create more fork 300-800max memory recovery, many cache problem time periods are reclaimed, memory has to be expanded, and there seems to be something wrong with the dynamic model. I don't know what so many processes in fork do.

The above conclusion looks very vague, but it provides a more comprehensive on-site summary and confirms the direction of the problem. Sure enough, after re-combing the code, the project team found a high concurrency scenario, and after modifying the parameters, the problem was solved.

This article mainly analyzes the relevant knowledge points of the example analysis of the sys toolbox, the content is detailed and easy to understand, the operation details are reasonable, and has a certain reference value. If you are interested, you might as well follow the editor to have a look, and follow the editor to learn more about the sample analysis of the sys toolbox.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.