Shulou(Shulou.com)06/01 Report--

[this article is transferred from Java Notes author: Xiaoqi 2012 original link: http://tech.it168.com/a2018/0629/3212/000003212205.shtml]

Preface

It has been about 10 days since the last time I was attacked by DDOS. I can't remember the specific day since the last time, and every time it came to nothing. However, with a relatively long-lasting attack recently, I think it is necessary to calm down and share the experience of being hacked.

Before describing the experience, let's briefly introduce the server configuration:

ECS 1 core 2G memory 1MB bandwidth, Linux system

RDS 2-core 240MB memory with a maximum of 60 connections

Redis 256MB shares instances, which is not used after moving.

CDN pays by quantity and caches small files.

The above configuration should be more than enough for a website with thousands of visitors a day, and about a dozen are supported. Here is a simple deployment of the website:

Well, the story begins, on June 14, 2018, I received an alarm from Aliyun system at 02:30, informing me that the website was inaccessible, but I was still asleep at that time.

As usual, I woke up at about six o'clock and habitually looked at my phone when another text message alarm came. In normal times, I can sleep for another two hours, but at this time, I am excited and sleepy at all. I am a blogger with thousands of visits anyway.

So, quickly get up and open the computer, try to visit the blog and forum, sure enough, the browser has been spinning around.

Problem troubleshooting

Try to log in to the server remotely:

Check Nginx and PHP-FPM,ps-ef | grep xxxx

Check the remaining memory of the system free-m

View CPU usage top

View the Nginx error log tail-f error.log

View log capacity ll-h

Check the number of concurrent connections netstat-nat | grep ESTABLISHED | wc-l

After a coquettish operation, there are no exceptions, memory and CPU are stable, and Nginx and PHP processes are fine. Then restart PHP and Nginx respectively, the website can also be accessed at first, and the home page of the community is stuck.

Check the error log, scan the log vigorously in the background, casually check a few IP, there are India, the United States, the Philippines and so on, of course, most of them are domestic IP. I actually brushed hundreds of trillion logs in one night (last time I cleaned it by D), anyway, I think it's a lot, compared to the usual number of visits to the site.

There have been several attacks before, but they are all in twos and threes. Just use Nginx to ban IP. This time, however, obviously not banning IP can solve the problem, so much IP collection is a problem (of course, can be obtained through regular matching), and may cause accidental injury.

On the way to work

However, going to work is the business, thinking for a while will not solve the problem, glanced at the error log, still hard to brush, and then casually posted a moments and then went to wash:

I murmured all the way, wondering if it was 9 o'clock when they got off the night shift on time and then they could visit normally and explain themselves.

At work

When we get to the company, the first thing of course is to log in to the server remotely and take a look at it. The error log is still brushing hard. Normally, this is a point in time and no user will access it.

Restart the service many times, visit the home page and get stuck, then instantly paralyzed, the entire site (community + blog) can not be accessed. In that case, you'd better go to work and wait for the attack to stop.

During the period, the friends in the group asked what was wrong with the website and couldn't open the coconut. Near noon, I checked the error log, and several IP tried to request a different address again, which was not a good thing at a glance, so I made a decisive deny. In other words, now that there are not so many requests, restart some Nginx and PHP processes, visit the home page or get stuck? What a strange egg.

Wondering if it is the problem with the RDS database, I checked the monitoring alarm panel, CPU and memory utilization and the current total number of connections are normal, nothing unusual, there are indeed fluctuations around 2: 00 in the morning-6: 00, but will not be killed by D. Now that you're all logged in, why don't you restart ECS and RDS by the way.

Sure enough, the restart was miraculously good, and I visited it with my cell phone when I was having lunch. It was normal and I could eat with peace of mind.

Problem solving

In fact, I am not clear about how the problem was solved in the end. I would like to say a few puzzling points:

ECS server CPU and memory are also at normal threshold

Both Nginx and PHP-FPM processes have been restarted

Although the number of RDS database connections fluctuates, it is not full and unreleased.

See, the error log requests come from hundreds of different IP, and most of them are URL from the visiting community.

And why are all these broilers at night? Cheap at night? Or organizing attacks in the Western Hemisphere?

Is it targeted or random this time? I hope it's random.

Stopped once in the middle of the community, the blog can always be accessed normally, suspected to be the home database query problem, based on the number of connections should not be this problem, is it the Bug of Discuz? But later, after restarting the database, it was able to access it normally.

In fact, Aliyun has basic DDOS protection to clean trigger values:

Request traffic per second: 300m

Messages per second: 70000

For ordinary small stations, it is absolutely impossible to reach the traffic threshold of 300m, and the peak CDN of the blog is only 3m.

Therefore, the attacks of these wavelet streams can only be borne silently by themselves, while the configuration of the machine is not high, and the bandwidth can only be bought freely to attack Sahuan. It is better to shut down the station directly and throw him a Nginx + static page to let it go.

Offensive and defensive strategy

If someone really D your site, you really have no way, of course I said the group is for small and medium-sized webmasters, you even DDOS basic protection cleaning threshold can not reach.

If you are just an unknown station, you don't need to think so much at all. Although the cost of DDOS is very low now, no one can get up early without profit, unless you offend someone.

Of course, we can not sit idly by for general attacks. Here are a few tips to share with you. Reverse proxy uses openresty.

Nginx optimization

Nginx is known as the maximum concurrency of 5W, in fact, for small and medium-sized sites, dozens or hundreds of concurrency is good, the most basic parameters can meet the needs. But for the sake of security, we'd better hide its version number.

# hide the version to prevent known vulnerabilities from being exploited server_tokens off; # configure in the http module

PHP optimization

The header information of the web page rendered by php will include the version number of php, such as X-Powered-by: php/5.6.30, which is a bit unsafe. Some hackers may scan for low-version php servers and exploit php vulnerabilities (such as hash conflicts) to attack the server.

# hide the version to prevent known vulnerabilities from being exploited php_admin_ [expose _ php] = off

IP blacklist

Blacklisting is a good way to deal with the most low attacks, or someone else's AB will crush you to death:

# add the following configuration to the http module of Nginx to deny 61.136.197.xxxscape # disable the IP segment deny 61.136.197.0Universe 24

Daily visits to IP

Limit the number of daily visits to a single IP. Normally, a user rarely has more than 10 access depths, and the bounce rate is generally between 50% and 70%. In fact, what we need to do is to limit the daily visits of a single IP to less than 100 or even 50.

Limit the number of concurrency

It is not enough to limit the number of visits. Attackers may instantly pour in hundreds of requests. If these requests go to the back-end service, it will destroy the database service, so we have to set the number of concurrent requests based on our own website visits.

Limit the number of concurrency of a single IP

Limit the total number of concurrency

It is recommended that you use leaky bucket algorithm to limit traffic to reshape traffic requests.

Configure CDN

Based on the consideration of bandwidth and normal user access speed, it is recommended to configure CDN. The following is the blog traffic usage, peak 3MB, which must be irresistible for my 1MB bandwidth server, and there is also community access.

Configure cach

Database resources are valuable, so try not to let requests go directly to the back end. In fact, blogs and communities have been configured with redis caching before the move. Because the previously purchased Redis service was a proprietary network, the new account could not be connected, and then gave up.

It seems that this time, it needs to be configured on the idle server, anyway, idle is also idle, it is good to be able to play a lost role.

Aliyun Redis accelerates Discuz Forum visit

Aliyun Redis accelerates Typecho blog visit

Summary

As mentioned earlier, for attacks, there is really no solution to the small station, so you can do a good job in basic protection. But for those broilers or those who are about to become broilers:

Software vulnerabilities must be patched in time and pay close attention to the relevant developments of the Internet.

Hackers use the hacked router to obtain network traffic to control Dalian broilers.

Most broilers have no security awareness and have been used for a long time. It has been found that many of them are cloud service provider hosts and hosting server hosts, which are controlled by hackers.

DDoS hacker attacks are changing to industrialization and platform services. if someone wants to hurt you, you can attack for a whole month with a button and a few hundred yuan, and then give yourself a song "cool".

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.