Shulou(Shulou.com)06/02 Report--

The bandwidth of IDC computer room has suddenly increased from 100m to 400m. Please analyze the problem and solve it.

[analysis of problems] 1) there are many reasons why IDC bandwidth is full. The common ones are:

a. Actually suffered from DDOS*** (encountered several times, the impact is rare, including cases of extortion).

b. Internal server poisoning, a large amount of outgoing traffic (this problem the old boy answered the alarm more than 5 times)

c. Website elements (such as pictures) are stolen and promoted on the portal page, resulting in a large amount of traffic (more than 3 times)

d. Cooperative companies come to capture data, for example: provide API data interface to cooperative companies (friends of cooperative companies know this)

e. After buying the CDN business, CDN grabbed the origin server (this is also a lot of times).

f. There are other reasons that should not be mentioned if they are not common.

2) the CDN bandwidth is abnormal, but the origin server is not abnormal.

This kind of problem is basically caused by frequent access to the data cached in CDN. The solution can be found in the final case.

3) the CDN bandwidth is abnormal, and so is the origin server.

It may be due to the company's promotion, a large number of data access, and incomplete hot data in cache. Or CDN problems lead to data back-to-origin (the experience of CDN back-to-origin rate and improving the back-to-origin rate will be shared with you later). The impact is high bandwidth, high pressure on back-end static servers and images and storage.

(for the solution, see the old boy's 7-tier portal architecture case article http://oldboy.blog.51cto.com/2561410/736710)

[problem solving]

The possible causes of the problems are analyzed so that they can be compared and investigated.

a. Really suffer from DDOS***

The solution to the DDOS problem the old boy has written an original article (http://oldboy.blog.51cto.com/2561410/845349), providing 17 ideas for your reference, so it is not the most common one that suffers from real DDOS*** and has an impact on it.

b. Internal server poisoning, a large amount of outgoing traffic.

The solution to this problem is relatively simple, some friends may say, look at the server traffic, which machine with high bandwidth will be fine. In fact, the actual solution is much more complicated than this, the bandwidth is full, all monitoring can not be seen.

A better idea is to contact the computer room to make sure that there is no problem with the server room (it is generally unable to help us), ask the computer room to disconnect the network cable from the external IP server, such as the load balancer, and keep only × × SERVER, and then disconnect the line of the internal server out of the network light gate and cut off the source of outgoing traffic.

Next, check the monitoring traffic service, determine the server of outgoing traffic, and then process it.

In fact, the occurrence and rapid positioning of this problem has a lot to do with the operation and maintenance norms and systems of many companies. when the old boy shared the operation and maintenance training for some companies, he found that the problem was very serious (the appearance was very good. Internal operation and maintenance standards, systems lack a lot), we all discussed very in-depth, the actual use is still a gap with the chat.

For example, some companies develop direct FTP connections to release code at any time, or developers are responsible for regularly launching multiple times. And the operation and maintenance staff do not know, the result leads to the location of the problem for a long time, which suggests that most of the companies think about it.

The old boy's idea of operation and maintenance is that if the website computer room is compared to a house, the first thing is to block the back door (inside), and the second is to monitor the front door (do a good job of security, leave a small window for outsiders to see, that is, port 80 service, and arrange to stand guard on duty at the same time).

The endless release of code at any time of the website is crucial to the stability of the site. It is also critical for the operation and maintenance personnel to locate the fault. According to the incomplete survey of the old boy, more than 50% of the important operation and maintenance failures are caused by the program code. this is also the reason why the old boy inculcates the suggestion CTO when he does training and sharing with the enterprise, and assigns more responsibility for the stability of the website to the developer rather than the operation and maintenance staff. If this idea is not reversed, the instability of the website will be difficult to change.

c. Website elements (such as pictures) have been stolen.

This belongs to the basic optimization of the website, apache,lighttpd,nginx has a hotlink protection program, must be done. Speaking of this, a student of the old boy went to work in an enterprise and found that there was no hotlink protection on his website. as a result, he did not make it known to the boss, so he did it directly, and then happily left a message to me at that time, saying that he had done hotlink protection for the company. Very successful, resulting in the company's foreign cooperation business, are all small forks, fortunately found in time did not have a big problem.

Dmure. Cooperation companies come to capture data, such as providing API data interfaces to partners or purchasing CDN services.

The most common is to purchase CDN services, such as CDN to create a new node (maybe dozens of machines), and directly come to our IDC to capture data (some do better at night). The traffic caught by the original station skyrocketed, seriously leading to service downtime. Several CDN companies have had this problem. I hope CDN can see that it can be improved, after all, the user God.

Of course, cooperation with telecom, Unicom, GOOGLE,BAIDU, Ciba and other companies will also have high traffic, including the problem of crawling data for cooperative search engines. Sometimes although the bandwidth traffic is not high, but the server or database can not support, search engines specifically like to climb our site search, DISCUZ,CMS and other early open source programs search is the site-wide like%% way to search the database, a few crawlers came, directly hung up, of course, this is not discussed in this article, the solution will talk later.

f. There are other reasons that should not be mentioned if they are not common.

The above points are more common, but other reasons are rare, so it's not easy to type so many words.

Example 7: analyze the photo service log, rank the log (the sum of the image sizes of each image visit), and take top10, that is, calculate the total access size of each url

Description: example 7 of the production environment application: this function can be used for IDC and CDN website traffic bandwidth is very high, and then by analyzing which elements of the server log occupy too much traffic, and then optimize and cut the picture (see the old boy's "Taobao Singles Day super traffic response article comments"), compression js and other measures.

This question needs to output three metrics: [number of visits] [number of visits to a single file size] [file name (with URL)]

Answer:

test data

59.33.26.105-[08/Dec/2010:15:43:56 + 0800] "GET / static/p_w_picpaths/photos/2.jpg HTTP/1.1" 11299 "http://oldboy.blog.51cto.com/static/web/column/17/index.shtml?courseId=43"" Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152) .net CLR 3.5.30729) "59.33.26.105-[08/Dec/2010:15:43:56 + 0800]" GET / static/p_w_picpaths/photos/2.jpg HTTP/1.1 "11299" http://oldboy.blog.51cto.com/static/web/column/17/index.shtml?courseId=43"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) .net CLR 3.0.4506.2152; .NET CLR 3.5.30729) "59.33.26.105-[08/Dec/2010:15:44:02 + 0800]" GET / static/flex/vedioLoading.swf HTTP/1.1 "3583" http://oldboy.blog.51cto.com/static/flex/AdobeVideoPlayer.swf?width=590&height=328&url=/`DYNAMIC`/2"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727) .NET CLR 3.0.4506.2152 .net CLR 3.5.30729) "124.115.4.18-- [08/Dec/2010:15:44:15 + 0800]" GET /? = HTTP/1.1 "200 46232"-"-" 124.115.4.18-- [08/Dec/2010:15:44:25 + 0800] "GET / static/js/web_js.js HTTP/1.1" 4460 "-"-"124.115.4.18-- [08/Dec/2010:15:44:25 + 0800] "GET / static/js/jquery.lazyload.js HTTP/1.1" 1627 "-"-"method 1: calculate through two arrays

Because the final result we want is the number of visits to a file and the traffic consumed, consider setting up two arrays indexed by the file name, one storing the access times and the other saving the consumed traffic, so that when traversing the file line by line using awk, the number of times array is + 1, and the file size of the traffic array is accumulated until the file scanning is completed. By traversing the output two arrays, you can get the number of counter-questions and the total traffic consumption of the file.

[root@locatest scripts] # awk'{array_num [$7] +; array_size [$7] + = $10} END {for (x in array_num) {print array_ size [x], array_num [x], x}} 'access_2010-12-8.log | sort-rn-k1 | head-10 > 1.log method II: [root@locatest scripts] # awk' {print $7 "\ t" $10} 'access_2010-12-8.log | awk' {S [$1] + = $2 S1 [$1] + = 1} END {for (i in S) print S [I], S1 [I] I}'| sort-rn | head-10 > 2.log [root@locatest scripts] # diff 1.log 2.log [root@locatest scripts] # cat 1.log57254 1 / static/js/jquery-jquery-1.3.2.min.js46232 1 /? = 44286 1 / / back/upload/course/2010-10-25-23-48-59-048-18.jpg33897 3 / static/p_w_picpaths/photos/2.jpg11809 1 / back/upload/teacher/2010-08-30-13-57-43- 06210.jpg10850 1 / back/upload/teacher/2010-08-06-11-39-59-0469.jpg6417 1 / static/js/addToCart.js4460 1 / static/js/web_js.js3583 2 / static/flex/vedioLoading.swf2686 1 / static/js/default.js

The above is reproduced in Director Li, thanks to Director Li for his hard work.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.