Shulou(Shulou.com)06/01 Report--

Recently, I found a problem when I was working on a project. When the website was opened, it stopped and turned around. About 5 minutes later, I went back to the 504 gateway and timed out.

At that time, the first reaction was that there was something wrong with my network architecture, so I looked down from the front-end LVS. My architecture was LVS+NGINX+TOMCAT, and there was no problem up to nginx. I couldn't see anything from the error log of nginx. I was thinking at that time that the bad gateway thought that nginx could not find the port of tomcat when it threw the request to tomcat.

[root@web1 ~] # netstat-lntup | grep 800

Tcp 00: 8000: * LISTEN 4208/java

Tcp 00: 8002: * LISTEN 4208/java

I don't see any problem, but I find it when I compare it with the online server.

[root@web1 ~] # netstat-lntup | grep 800

Tcp 00: 8000: * LISTEN 4208/java

Tcp 00:: ffff:127.0.0.1:8001: * LISTEN 4208/java

Tcp 00: 8002: * LISTEN 4208/java

It is found that tomcat is missing a port, why does it not cause this problem?

Check the tomcat log and find that tomcat is not up and has been stuck in the connection to the database.

So I checked the mysql port on the web side and found that I could get through.

[root@web2 server] # telnet 10.10.0.126 3306

Trying 10.10.0.126...

Connected to 10.10.0.126.

Escape character is'^]'.

^]

Telnet >\ Q

Connection closed.

Come here and find that your mind is broken!

The twists and turns, inadvertently check the mysql port on db2, found that the result is not the same as on web

[root@db2 root] # telnet 10.10.0.126 3306

Trying 10.10.0.126...

Connected to 10.10.0.126.

Escape character is'^]'.

N

5.5.32-log0pKhJw=db! > MIgIxAwW,s "mysql_native_password

Why is that?

So I recalled a little bit, thought about the difference between web and db, and finally found the problem.

10.10.0.126 and 10.10.0.127 are the read vip and write vip of my data. Only these two machines have the ip of 10.10.0.0 network segment.

There is only one public network ip on web, so route-n looks at the gateways of web and db

[root@web1 server] # route-n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

202.192.168.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2

169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth2

0.0.0.0 202.192.168.1 0.0.0.0 UG 0 0 0 eth2

[root@db2 ~] # route-n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

202.192.168.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

10.10.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth0

0.0.0.0 202.192.168.1 0.0.0.0 UG 0 0 0 eth0

Obviously, the gateway for 10.10.0.0 is missing on web

So a gateway is added on the web side.

Route add-net 10.10.0.0 netmask 255.255.255.0 dev eth2

Check again.

[root@web1 server] # route-n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

202.192.168.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2

10.10.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2

169.254.0.0 0.0.0.0 255.255.0.0 U 1002 00 eth2

0.0.0.0 202.192.168.1 0.0.0.0 UG 0 0 0 eth2

It is the same as the DB side.

Check the mysql port

[root@web1 server] # telnet 10.10.0.126 3306

Trying 10.10.0.126...

Connected to 10.10.0.126.

Escape character is'^]'.

N

5.5.32-log ±vj3 [skyscraQ {9H=9E0] fjamysql_native_password

It worked!

Restart tomcat to view the tomcat port

[root@web1 server] # netstat-lntup | grep 800

Tcp 00: 8000: * LISTEN 4208/java

Tcp 00:: ffff:127.0.0.1:8001: * LISTEN 4208/java

Tcp 00: 8002: * LISTEN 4208/java

The port is up.

Page access is also normal!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.