Shulou(Shulou.com)06/01 Report--

Since there are many links involved in this performance tuning project, it is best to monitor all the content in the generation environment, taking into account the lowest overhead, so that it can be carried out from both the application server and the database server, using nmon as the monitoring basic data, and monitoring JVM and database alarms and snapshots at the same time.

All monitoring contents are means, and only regular and meaningful data from massive monitoring logs is the basis of performance optimization. With the data is the analysis of the data, this article will first introduce the data that need to be obtained, and the content will also be my experience from the project.

Basic environment:

Two database servers, do the database cluster. Application server-JVM thread

Tongweb is mainly used in the project (the old system version is very low), and the monitoring content is similar to the following:

Monitor content... "2018-01-11T02:25:55.663+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "NumConnCreated", "10", "2018-01-11T02:25:55.663+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "NumConnAcquired", "111292", "2018-01-11T02:25:55.663+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor" Server=server "," NumConnNotSuccessfullyMatched "," 0 "," 2018-01-11T02:26:25.670+0800 "," com.tongtech.tongweb:type=jvm,category=monitor,server=server "," UpTime "," 222520621 "," 2018-01-11T02:26:25.670+0800 "," com.tongtech.tongweb:type=jvm,category=monitor,server=server "," HeapSize "," 2143485952 "," 2018-01-11T02:26:25.671+0800 "," com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server "," NumConnUsed "," 0 " "2018-01-11T02:26:25.671+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "NumConnSuccessfullyMatched", "0", "2018-01-11T02:26:25.671+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "WaitQueueLength", "0", "2018-01-11T02:26:25.671+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "NumConnDestroyed" "0", "2018-01-11T02:26:25.671+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "ConnRequestWaitTime", "4", "2018-01-11T02:26:25.672+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server", "NumConnFailedValidation", "0", "2018-01-11T02:26:25.672+0800", "com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server" NumConnReleased, 111292, 2018-01-11T02:26:25.672+0800, com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server, NumConnFree, 10,... Focus on content

The monitoring data of tongweb obtains information such as the status of the connection pool. Our method is to convert the log into readable data by means of Excel macros, and carry out graphic analysis. The details will be explained separately.

JVM thread monitoring instructions

Monitoring significance

Through the JVM monitoring of tongweb, we can initially determine whether the performance peak time point and connection pool are full, and further determine whether the performance bottleneck during the connection peak period appears in the application, which is particularly important for the future performance analysis. We can classify the main performance problems and reduce unnecessary work.

Application Server-netstat

In the Internet RFC standard, Netstat is defined as: Netstat is a program that accesses network connection status and related information in the kernel. It can provide TCP connection, TCP and UDP snooping, and related reports of process memory management.

Monitoring content

The following is the log excerpt obtained in the project

.. Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0 Proto Recv-Q Send-Q Local Address Foreign Address State tcp 2049 0.0.0.0 Proto Recv-Q Send-Q Local Address Foreign Address State tcp * LISTEN tcp 0 0 0.0.0.0 Proto Recv-Q Send-Q Local Address Foreign Address State tcp 139 0.0.0.0 LISTEN tcp 0 0 0 LISTEN tcp 427 0 0 0 LISTEN tcp 0 0 0 111 0 0 0 LISTEN tcp 0 0 0 5 44 0 0 0 LISTEN tcp 0 0 0 LISTEN tcp 0 0 0 LISTEN tcp 631 0 0 0 * LISTEN tcp 0 0 127 0 0 0 1 V 25 0 0 0 V * LISTEN tcp 0 0 0 5 0 0 0 LISTEN tcp 0 0 0. 0 669 0. 0 0. 0. 0. Application Server-nmon

As the main analysis means of this performance optimization, nmon plays a particularly important role. The following is the explanation of wiki, which can be understood in time.

Nmon collects the following operating system statistics:

CPU and CPU threads Utilisation

CPU frequency for servers or virtual machines that can alter their clock rate

GPU stats including utilisation, MHz and temperatures

Physical and Virtual Memory use

Disk read & write and transfers

Disk Groups decided by the user

Swap and Paging

Network read & write and transfers

Local File-systems

Network File-system (NFS)

Top Processes by CPU use, Memory size and I/O rates

Kernel stats including Run Queue, context-switch, fork, Load Average & Uptime

Large and Huge memory pages

Virtual Machine stats (depending on the hardware)-useful for Linux running KVM to host virtual machines

Resources in the Server and virtual machine

In fact, nmon is more like a snapshot of system performance overhead, combined with the analysis tools of nmon can clearly grasp the indicators of the system.

Download analysis tools

Database server-alarm

Understanding the database alarm log is also a key link to grasp the current performance.

The log is as follows. If there is an error, you can analyze and solve it according to the specific situation.

2018-01-11-00.36.36.090562 I13363168A459 LEVEL: ErrorPID: 2228842 TID: 142490 PROC: db2syscINSTANCE: db2 NODE: 000 DB: TRADEEDUID: 142490 EDUNAME: db2agent (* *) 0FUNCTION: DB2 UDB, Query Gateway, sqlqg_fedstp_hook, probe:40MESSAGE: Unexpected error returned from outer RC=DATA # 1: Hexdump, 4 bytes0x07000007053F28D0: 8126 0012. &. Database Server-Snapshot

The snapshot of the database log will be used as the main basis for analysis, in which you can analyze the cost of database time, as follows:

Number of automatic storage paths = 1Automatic storage path = / db2data Node number = 0 State = In Use File system ID = 9223372079804448776 Storage path free space (bytes) = 69730709504 File system used space (bytes) = 139648946176 File system total space (bytes) = 209379655680.

This article only lists the methods of analysis, and I will sum up the specific operation slowly when I have time.

The use of tools is important, but performance tuning is more than that. You must prepare for a long-term battle step by step.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.