Shulou(Shulou.com)06/01 Report--

The following day, the website was hit by traffic *. The nagios alarm only indicated that the system was overloaded, and then opened the traffic map to have a look. Hey, Nima, shit, how can this be? The flow rate is beyond metering!

Then connect to the website server, connect for a long time to connect, after connecting, casually hit a ls command! This is pure habit! Then it got stuck, and it took me a long time to react! Then top for a while, also waiting for a long time! The system load is off the meter. Load soared to more than 100, nagios can not get the status! Because of the heavy load of the system, the nrpe message can not be sent out!

Then without hesitation, stop the apache,java process directly and start it again! Free up the space occupied by the program! The load of the system process is more than 100, how do you expect to deal with errors? It took a long time to react to ls and told the superior to stop the service. Wait for the reply, maybe the server will be dead!

Then I looked at the apache access log. At that time, I looked directly at the IP accessed by apache! See a lot of visitors with suffixes! Like 1.1.1.1.ha-dc 1.1.1.1.index-bn-spind-cn, what kind of messy IP?

Then directly match the keywords on the firewall to block it!

Iptables-I INPUT-m string-algo bm-string "1.1.1.1.ha-dc"-j DROP

Just discard it, and if it is set to reject, the system will also give a reply value over there.

And then sealed off those IP like this.

After staring at top for a while, I found that the java program was not normal, and the displayed cpu usage often fluctuated at about 500% and 800%. Our environment is middleware made of weblogic. Java programs are generally in the normal range of 100%, 200%. But it's definitely not normal to be so high.

Then directly top-H point of view to take up the largest resources, sure enough, are java! Then I found the thread with the longest running time and found that there was no exception, and there was no conversion to hexadecimal as mentioned on the Internet.

Ulimit-a takes a look at the largest thread allowed for java. Found that the setting is a little low.

Only 65535 colleagues are allowed, it may be this problem!

Then set the number of threads to the maximum.

Ulimit-n 102400

Java allows you to set the maximum thread!

Then after a while, java stabilized!

Er, (⊙ o ⊙) (⊙ o ⊙) Java can still have an exception! I am speechless.

Then at this point, the program exception is resolved. But the traffic that the website is brushed has not been solved! The system load is fixed at about 6, looking at the apache access log. Found that some of their IP can still pass! Uh,

(⊙ o ⊙) (⊙ o ⊙) (⊙ o ⊙)

Because the number of IP visits is too large, iptables can not block all of them, so I asked the boss for a hardware firewall.

And then set it up in the hard defense. So far, the problem is solved!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.