Shulou(Shulou.com)06/01 Report--

It is believed that many inexperienced people have no idea about how to analyze and install systemtap and flame diagram in Linux system. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

SystemTap is a tool for monitoring and tracking the Linux kernel, which is somewhat similar to the old technology of DTrace, because SystemTap does not use tools to build a special kernel, so it allows us to dynamically install modified tools when we use them.

1.1 We need to check the operating system version before installation. The command is as follows:

# uname-r

3.10.0-327.el7.x86_64

# cat / etc/RedHat-release

CentOS Linux release 7.2.1511 (Core)

I use CentOS 7.2 here, and the installation method described below is also on CentOS.

1.2 the following two packages are required to install and deploy SystemTap for SystemTap:

Systemtap

Systemtap-runtime

Under root authority, install using yum, and the command is as follows:

Yum install systemtap systemtap-runtime

Before using SystemTap, you need to install kernel packets. In general, run the following command to install it:

Stap-prep

After running the command, it often fails, and the installation is not successful. After the installation fails, you can install the kernel package manually.

My server reported the following error:

No package kernel-debuginfo-3.10.0-514.el7.x86_64 available. Error: Nothing to do Loaded plugins: fastestmirror, langpacks enabling epel-debuginfo Loading mirror speeds from cached hostfile Could not find debuginfo for main pkg: kernel-3.10.0-514.el7.x86_64 No debuginfo packages available to install package kernel-debuginfo-3.10.0-514.el7.x86_64 is not installed problem installing rpm (s) kernel-debuginfo-3.10.0-514.el7.x86_641.3 the installation packages required to manually install kernel packages are as follows:

Kernel-debuginfo

Kernel-debuginfo-common

Kernel-devel

You need to install an installation package that is consistent with your kernel version, and run the command to view the version:

# uname-r

3.10.0-327.el7.x86_64

You can specify that your kernel is 3.10.0-327.el7.x86_64, so the packages you need are as follows:

Kernel-debuginfo-3.10.0-514.el7.x86_64.rpm

Kernel-debuginfo-common-x86_64-3.10.0-514.el7.x86_64.rpm

Kernel-devel-3.10.0-514.el7.x86_64.rpm

Important! Note: these three packages must match your kernel version, otherwise the installation will not be successful.

Download these three rpm packages and go to http://rpm.pbone.net to download them. Or the simplest way to search for package names in google.

Install the three packages separately:

Rpm-ivh kernel-debuginfo-common-x86_64-3.10.0-514.el7.x86_64.rpm

Rpm-ivh kernel-debuginfo-3.10.0-514.el7.x86_64.rpm

Rmp-ivh kernel-devel-3.10.0-514.el7.x86_64.rpm

You can also try using yum installation, so I won't repeat it here.

1.4 Test whether the installation and deployment is successful, and how to test it? Run the following command:

Stap-v-e 'probe vfs.read {printf ("read performed\ n"); exit ()}'

The deployment failure prompt is as follows:

# stap-v-e 'probe vfs.read {printf ("read performed\ n"); exit ()}' Pass 1: parsed user script and 119library scripts using 220240virt/33264res/3272shr/30276data kb, in 320usr/20sys/717real ms. Semantic error: while resolving probe point: identifier 'kernel' at / usr/share/systemtap/tapset/linux/vfs.stp:882:18 source: probe vfs.read = kernel.function ("vfs_read") ^ semantic error: missing x86 * 64 kernel/module debuginfo [man warning::debuginfo] under' / lib/modules/3.10.0-514.el7.x86 * 64max build' semantic error: while resolving probe point: Identifier 'vfs' at:1:7 source: probe vfs.read {printf ("read performed\ n") Exit ()} ^ semantic error: no match Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 234228virt/47084res/5204shr/42164data kb, in 110usr/190sys/899real ms. Missing separate debuginfos, use: debuginfo-install kernel-3.10.0-514.el7.x86_64 Pass 2: analysis failed. [man error::pass2] the prompt for successful deployment is as follows:

# stap-v-e 'probe vfs.read {printf ("read performed\ n"); exit ()}' Pass 1: parsed user script and 119library scripts using 220244virt/33264res/3272shr/30280data kb, in 360usr/30sys/530real ms. Pass 2: analyzed script: 1 probe, 1 function, 4 embeds, 0 globals using 351120virt/165268res/4472shr/161156data kb, in 1700usr/440sys/3951real ms. Pass 3: translated to C into "/ tmp/stapeOM1BT/stap_bbe80ab58c56cba7fc7b6efecaadacb6_1647_src.c" using 351120virt/165564res/4768shr/161156data kb, in 10usr/50sys/63real ms. Pass 4: compiled C into "stap_bbe80ab58c56cba7fc7b6efecaadacb6_1647.ko" in 1880usr/490sys/4598real ms. Pass 5: starting run. Read performed Pass 5: run completed in 10usr/40sys/406real ms. II. Introduction and installation of flame diagram 2.1.Introductionofficial website: flame diagram (FlameGraphs)

The flame diagram (Flame Graph) was invented by Brendan Gregg, a master of Linux performance optimization. It is an artifact for locating difficult and complicated problems, such as high CPU usage, memory leaks and so on.

Vertically represents the depth of the call stack and horizontally represents the time consumed. Because the call stack is sorted alphabetically horizontally, and the same call stack is merged, the larger the width of a grid, the more likely it is to be a bottleneck.

To sum up, it is mainly to look at those flames that are relatively broad, and pay special attention to those similar to those in Pingdingshan.

Example of a flame chart:

2.2.The installation of the flame image is relatively simple. You only need to down the script from github. The command is as follows:

Git clone https://github.com/brendangregg/FlameGraph

2.3 generating a flame map generally requires the following steps:

1), grab and capture stack information: use tools such as perf/systemtap/dtrace to grab the running stack information of the program.

2) folding stack: use the stackcollapse program in FlameGraph to analyze and combine the captured information; the trace tool grabs the stack information of each time the system and the program is running, and needs to analyze and combine them to accumulate the repeated stacks together, thus reflecting the load and critical paths.

3) generate flame diagram: use flamegraph.pl in FlameGraph to analyze the stack information output from stackcollapse to generate flame diagram

Stackcollapse has many tools to analyze the kernel information captured by different trace tools, such as stackcollapse-stap.pl is used to analyze the information captured by SystemTap.

Third, use specific examples to illustrate how SystemTap and FlameGraph are used.

3.1The sample_bt_vfs tool was written by Chun GE, the author of openresty. This tool samples the user space call stack on the virtual file system (VFS) in order to render the file Imax O flame diagram, which can accurately reflect the distribution of file Imax O data volume or file Imax O delay in different user space code paths in any running user process.

Details of the use of the tool: sample_bt_vfs

Toolkit download address: openresty-systemtap-toolkit

Method: git clone https://github.com/openresty/openresty-systemtap-toolkit.git

The tools are placed in the openresty-systemtap-toolkit directory

Take viewing nginx as an example:

1) use SystemTap for kernel data acquisition:

Run:

$. / sample-bt-vfs-p 12345-T3 > a_vfs.log WARNING: Tracing 20636 (/ opt/nginx/sbin/nginx). WARNING: Time's up. Quitting now... (it may take a while) WARNING: Number of errors: 0, skipped probes: 2 parameters:

-p: represents the PID of the program

-t: represents the time when the data is to be collected

A_vfs.log: is the name of the file to save the collected data.

2) analyze the data collected by the combination

Run:

$. / FlameGraph/stackcollapse-stap.pl a_vfs.log > a_vfs.cbt3), generate flame diagram

Run:

$. / FlameGraph/flamegraph.pl. / a_vfs.cbt > a_vfs.svg here is an example of the "file I use O Flame Map":

A_vfs.svg is the generated flame diagram, and it is time-consuming to open it with a browser and find a wide and flat item.

3.2 sample-bt tool this script can sample the call stack of any user process you specify. The call stack can be user space, kernel space, or both. Its output is the summarized call stack (by total).

Tool details: sample-bt

For example:

. / openresty-systemtap-toolkit/sample-bt-vfs-p 41046-t 50-ku > 17_on_cpu.log

Parameters:

-p: represents the process ID

-t: indicates that the sampling time is in seconds

-u collect user space call stack information

-k collects kernel space call stack information

The rest of the flame diagram is generated in the same way as 3.1.

What is Linux system Linux is a free-to-use and free-spread UNIX-like operating system, is a POSIX-based multi-user, multi-task, multi-threaded and multi-CPU operating system, using Linux can run major Unix tools, applications and network protocols.

After reading the above, have you mastered how to analyze and install systemtap and flame diagram in Linux system? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.