Shulou(Shulou.com)06/02 Report--

Abstract: Linux servers often encounter some system and application problems, how to analyze and check, need sharp tools, below a summary list of some common tools, trace tool; finally enumerated the recent development of the hadoop community in the development of distributed system trace tool. Overview: referencing pictures in linux-performance-analysis-and-tools

Linux servers often encounter some system and application problems, how to analyze and troubleshoot, need sharp tools, below a summary list of some common tools, trace tool; finally enumerated the recent development of the hadoop community in the development of distributed system trace tool.

Overview:

Http://www.brendangregg.com/index.html

Http://www.slideshare.net/brendangregg/linux-performance-analysis-and-tools

Https://github.com/brendangregg/perf-tools/

Http://www.brendangregg.com/linuxperf.html

Refer to the pictures in linux-performance-analysis-and-tools to illustrate the location of these tool trial levels

Most of the tools mentioned in them, which have been used in my daily toolbox or in practical cases, are of high value. Here, it is convenient for you to index:

Nicstat: see here oprofile: see here systemtap: see here iotop: see here blktrace: see here dstat: see here strace: see here pidstat: see here vmstat: see here slabtop: see here tcpdump: see here mpstat: see here netstat: see here Tcprstat: see OS system commands here

System Information (RHEL/Fedora) uname-an or cat / proc/version # print system informationLinux hadoopst2.cm6 2.6.18-164.el5 # 1 SMP Tue Aug 18 15:51:48 EDT 2009 x86 "64 GNU/Linuxuptime15:42:46 up 674 days, 6 min, 35 users, load average: 1.30,5.97 11.53cat / etc/redhat-releaseRed Hat Enterprise Linux Server release 5.4 (Tikanga) lsb_releaseLSB Version:: core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarchcat / proc/cpuinfocat / proc/meminfolspci-list all PCI deviceslsusb-list USB deviceslast Lastb-show listing of last logged in userslsmod-show the status of modules in the Linux Kernelmodprobe-add and remove modules from the Linux Kernel commonly used commands / tools psTo print a process tree: ps-ejH / ps axjfTo get info about threads: ps-eLf / ps axmsulimit-alsof-list open files, UNIX everything file lsof-p PIDrpm/yumrpm-qf FILE # file rpm package rpm-ql RPM # rpm contains file / var/log/yum.log # yum update package log / etc/XXX # system-level program configuration directory For example, / etc/yum.repos.d/ yum source configuration / var/log/XXX # log directory, such as / var/log/cron # crontab log, you can view the scheduling execution ntpd-Network Time Protocol (NTP) daemon, synchronize the machine time in the cluster squid-proxy caching server, and the agent system of the cluster WebUI monitors mpstat-Report processors related statistics. Note that% sys% iowait value vmstat-Report virtual memory statisticsiostat-Report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions.netstat-Print network connections, routing tables, interface statistics, masquerade connections, and multicast membershipsnetstat-atpn | grep PIDganglia-a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids.sar/tsar-Collect, report, or save system activity information Tsar is Taobao's own improved version of timed sampling (per minute), can check the history (default 5 minutes), can make up for ganglia display more detailed information iftop-the "top" bandwidth consumers shown. Iftop wikiiotopvmtouch, Portable file system cache diagnostics and control network related telnet/nc IP PORT-confirm whether the destination port is accessible, only ping is accessible, not necessarily the port is accessible, and ifconfig/ifup/ifdown-configure a network interfacetraceroute-print the route packets trace to network hostnslookup-query Internet name servers interactivelytcpdump-dump traffic on a network may be prohibited by firewalls. Similar to open source tools wireshark, netsniff-ng, more tools to compare lynx-a general purpose distributed information browser for the World Wide Webtcpcp-allows cooperating applications to pass ownership of TCP connection endpoints from one Linux host to another one. Program / process related

Static message ldconfig-configure dynamic linker run time bindingsldconfig-p | grep SO to check whether so is ldd-print shared library dependencies in link cache, to check the sonm-list symbols from object files that exe or so depends on, but grep to find out whether there is a related symbol and whether Undefined.readelf-Displays information about ELF files. Elf-related information can be realised, such as 32amp 64-bit, applicable OS, processor dynamic information gdbcat / proc/$PID/ [cmdline | environ | limits | status |...]-process related information pstack-print a stack trace of a running processpmap-report memory map of a processjava related JDK Tools and UtilitiesJava Troubleshooting Toolsjinfo-print java process information, such as classpath,java.libary.path (jni so directory) jstack-print a stack trace of a running java process Jmap-report memory map of a java processjmap-histo:live can trigger full gcjmap-dump:live,file=$FILE dump heap memory for jhat and other tools such as debug to analyze the occupancy of object in heap jhat-Heap Dump Browser-Starts a web server on a heap dump file (eg, produced by jmap-dump), allowing the heap to be browsed. From http service, browsers access and view-J-mxXXXm, and you need to increase the heap size when analyzing large files. If the object data is too large or takes up too much memory, it is very likely that the memory leakMemory Analyzer (MAT)-eclipse plugin,Java heap analyzer visualization tool, but due to the limitation of machine memory, it is impossible to analyze too large heap dump filejdb-available services as server,eclipse and other tools to remotely connect and debug jstat-Java Virtual Machine Statistics Monitoring Tooljstatd-Virtual Machine jstat Daemon. Can cooperate with jvisualvmjvisualvm-Java Virtual Machine Monitoring, Troubleshooting, and Profiling Tool Jstatd/jmx can be connected remotely, visual display tool: demonstrate jvmtop-In a top-like manner, displays JVM internal metrics (e.g. Memory information) optimization article written by of running java processes.JVM performance optimization JVM developers OverviewCompilersGarbage collectionConcurrently compacting GCScalabilityHPROF-Heap Profiler: java-agentlib:hprofTrace/Debug/Profiling tool

General tools write log, but when the system is online or cannot source code, strace-trace system calls and signals example: strace/ltrace application example: can track system call time, such as machine cpu:%sys high problem% time seconds usecs/call calls errors syscall

--

67.90 3966.320849 496 7992161 3050250 futex

25.80 1507.326693 127093 11860 epoll_wait

BLK trace, generate traces of the iBank o traffic on block devicesltrace-A library call tracerxtracegprof-a performance analysis tool, sampling and call-graph profilingvalgrind-an instrumentation framework for building dynamic analysis tools. Automatically detect many memory management and threading bugs, and profile your programs in detailsystemtap-a simple command line interface and scripting language for writing instrumentation for a live running kernel plus user-space applications for complex tasks that may require live analysis, programmable on-line response, and whole-system symbolic access.Linux version of DTrace (developed by SUN on Solaris) is powerful, kernel, user-space app,cross language (java perl python ruby) Build-in markers (pg mysql) can write and reuse simple scripts to deeply examine the activities of a live systemData can be extracted, filtered, and summarized quickly and safely, to enable diagnoses of complex performance or functional problems rich "tapset" script libraryjava trace tools btrace-dynamic tracing tool for the Java platform. UserGuide is based on dynamic bytecode modification technology (Hotswap) to realize the tracking and replacement of run-time java programs. The implementation principle of BTrace is introduced in detail in byteman-simplifies tracing and testing of Java programs. Can modify a running application without needing to stop and restart it.define rules specifying the side effects you want to inject and BTrace class java syntax Distributed Tracing ToolsDapper, a Large-Scale Distributed Systems Tracing Infrastructurex-trace, a network diagnostic tool designed to provide users and network operators with better visibility into increasingly complex Internet applications.HTrace, a tracing framework intended for use with distributed systems written in javaAdd Tracing to HDFSUpdate HTrace for HBase

Linux observability tools | Linux performance observation tool

Linux benchmarking tools | Linux performance evaluation tool

Linux tuning tools | Linux performance tuning tool

Linux observability sar

Brendan Gregg is currently a senior performance architect at Netflix, where he does large-scale computer performance design, analysis, and tuning. He is the author of technical books such as Systems Performance and won the 2013 USENIX LISA Award for his work as a system administrator. Previously a performance leader and kernel engineer at SUN, he developed ZFS L2ARC at SUN and studied storage and network performance. He has also invented and developed a wave of performance analysis tools, many of which have been integrated into the operating system. His recent work includes research on the methodology and visualization of performance analysis, including the Linux kernel.

The above is a brief introduction to Gregg, and as mentioned in it, he shares a lot of Linux performance-related resources on his personal site, all of which are developed by himself:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.