Shulou(Shulou.com)05/31 Report--

This article shows you how to carry out slow fault analysis of web applications. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Not long after my friend worked as an operator on a shopping website, he called today to say that the front page opened slowly and the order could not be delivered normally, but there was no pressure to check the low utilization rate of CPU, but the memory was slightly higher than 86%, and the disk read and write was also very normal. Ask me how to solve it. I asked him some basic system architecture questions, knowing that the load balance done by F5 was distributed to the front-end 10 application servers, the middleware was JBOSS, and the back-end database was ORACLE. After sorting out the next train of thought, I decided to let my friend make the following analysis:

Log in to the front-end application server and first check and count the number of ip connections

[user@local ~] $netstat-ntu | awk'{print $5}'| cut-d:-F1 | sort | uniq-c | sort-n

one

1 Address

1 servers)

2 172.16.100.35

2 172.16.100.38

2 172.16.100.45

4 127.0.0.1

10 172.16.100.18

20 172.16.100.8

2048 172.16.100.98 # there are 2048 connections between server and load balancer

View JBOSS application port 8080

[user@local ~] $netstat-ntu | grep 8080

Tcp 00 172.16.100.26:8080 172.16.100.98:19593 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:16777 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:11913 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:1929 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:53641 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:43401 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:36233 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:28040 TIME_WAIT

Tcp 00 172.16.100.26:8080 172.16.100.98:5000 TIME_WAIT

...

...

...

[user@local ~] $netstat-ntu | grep 8080 | grep TIME_WAIT | wc-l

2003

As shown above, there are a lot of TCP waits between the front-end application server and the load balancer, which is the main reason for consuming memory and slowing down the front-end application. A large number of requests distributed to the application server by the load balancer may wait due to the processing bottleneck of JBOSS, and the actual processed requests are much smaller than the distributed requests.

2. Check the settings for the number of JBOSS connections

General middleware jboss connects to the back-end oracle, and there will be a file oracle-ds.xml under deploy. Let's take a look at the code that connects to the database:

Jdbc/TETD

Jdbc:oracle:thin:@172.16.100.18:1521:orcl

Oracle.jdbc.driver.OracleDriver

TETD_QT

Hpoui&bmn$m#e$m_n20uytwe@iil

twenty

fifty

Org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter

Note: set the value to 20 and the value to 50

3. Check the settings for the number of ORACLE connections

SQL > select count (*) from v$process-current number of connections

Display as 45

SQL > select value from v$parameter where name = 'processes'-the maximum number of connections allowed by the database

Display as 600

(note: when the maximum number of database connections is not enough, the client connection will fail intermittently and an error ORA-12519 will be reported)

Through the above data analysis, the number of connections displayed on the Oracle side does not reflect the due pressure, so where is the bottleneck? It should be that the connection pool parameter of JBOSS on JBOSS is set too small, so that the request card of the front end is located at JBOSS and cannot be connected to the back-end ORACLE through JBOSS, which leads to abnormal slow orders for front-end applications.

After the analysis, tell your friends to increase the JBOSS connection pool parameters and adjust them to 200,500. Later, after a friend adjusted the parameters of the JBOSS connection pool, the problem was significantly alleviated and the memory usage was also reduced.

This example tells us that the smallest part in the middle of a water pipe often determines the water flow through, so we should not forget to optimize the parameters of the middleware while optimizing the back-end ORACLE database.

Supplement 1. Modify the maximum number of ORACLE connections:

# modify the maximum number of oracle connections:

Alter system set processes = 1000 scope = spfile

Restart the database:

Shutdown immediate

Startup

Supplement 2. Modify / etc/sysctl.conf file

Net.ipv4.tcp_max_tw_buckets = 3000

Indicates that the system maintains the maximum number of TIME_WAIT sockets at the same time, and if this number is exceeded, the TIME_WAIT socket will be cleared immediately and a warning message will be printed. The default is 180000, which changes to 3000. This parameter controls the maximum number of TIME_WAIT sockets to prevent the server from being dragged to death by a large number of TIME_WAIT sockets.

The above content is how to analyze the slow failure of web application. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.