Shulou(Shulou.com)06/01 Report--

As an operation and maintenance engineer of a medium-sized website, I really encounter a website failure, seek ideal troubleshooting steps, my own experience, and add netizens' views.

The website is dead:

1, ping my website master IP, may be ping banned, not connected, may be the computer room network problems, then go to the ping computer room gateway!

2. If there is no problem with the computer room network, then I will go to see what the access is, if the server is abnormal or if the nginx reports an error.

Then I will check the hardware, my website is a simple nginx load + external firewall, then I will take a look at access. Log

Count the suspicious IP and behavior at this stage. If there is any, blacklist the suspect first.

3. Troubleshoot the route from the public network to our master station IP, tracert, whether it is possible to cross-domain problems, China Unicom network access failed? Or telecom? See if DNS has been hijacked.

4. At this time, I will take a look at the server. My website program is run by tomcat to see if the tomcat process is dead. Look at the log. Generally speaking, as long as there is no problem with the load, generally, http requests will not be piled on a server, which may be a load weight problem.

, or my tomcat (or other web container, memory setting problem)

In fact, these situations can be accomplished through zabbix monitoring (generally, if the number of visits increases sharply, or if there is a change in the front-end time, there is likely to be a shortage of cpu, and memory overflows occur in general programs. If system resources permit, increase the jvm size, initial stack, number of connections, or focus on development, on memory recovery).

5, you can try single sign-on to a node to see, encounter internal program call. Check the internal curl.

Or use httprequest to see which status code 200 accessed by post and get is OK.

The Great God's explanation: the best solution:

[senior] Didu-- big brother 21:54:06 on 2016-8-2

I will take a look at the monitoring first, because basically on the monitoring, I have done all these tests.

By monitoring the data, first narrow the scope of the investigation. Targeted to find the point of failure, troubleshooting. If you do this, it is estimated that the business has been interrupted for some time.

[senior] Didu-- big brother 21:55:54 on 2016-8-2

Respond quickly and minimize the impact first. It's up to you to do that.

[senior] Didu-- big brother 21:56:09 on 2016-8-2

The problem can be put aside first and the business can be restored first.

[senior] Didu-- big brother 21:56:23 on 2016-8-2

Business is the key, and problems can be checked slowly.

[senior] Didu-- big brother 21:56:41 on 2016-8-2

Because there are logs and monitoring data, you can slowly analyze exactly where the business interruption is caused.

[senior] Didu-Big Brother

When you take over the whole job, you should consider in advance how it can be restored immediately after the website is hung up, and the big company is the user's unaware recovery. Small companies may have a slight impact because of various restrictions.

[senior] Didu-- big brother 21:59:55 on 2016-8-2

By the time the website hangs up, you are going to check all kinds of problems, you are already late.

[senior] Didu-- big brother 22:00:56 on 2016-8-2

Personal opinion, for reference only.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.