Shulou(Shulou.com)06/01 Report--

Preface

Before sharing this event, I would like to take a look at CC***: with you.

[CC***]

  users use the proxy server to generate legitimate requests to the victim host, and the implementation of DDOS and camouflage is called CC (ChallengeCollapsar).

CC is mainly used for pages. The principle of CC is that some hosts keep sending a large number of packets to each other's server, causing server resources to be exhausted until the crash. CC is mainly used for * * pages. Everyone has this experience: when a web page is visited by a large number of people, it is slow to open a web page. CC is to simulate multiple users (the number of threads is the number of users) constantly visiting pages that require a lot of data operations (that is, a lot of CPU time), resulting in a waste of server resources. CPU is 100% for a long time. There will always be endless connections until normal access is aborted due to network congestion.

  CC is a kind of DDOS (distributed denial of Service), which seems to be more technical than other DDOSCC. You can't see the real source IP or a lot of abnormal traffic, but the server cannot connect normally.

Quote Baidu encyclopedia https://baike.baidu.com/item/cc%E6%94%BB%E5%87%BB/10959545?fr=aladdin

A sour moment

  one afternoon, when I was about to get off work, the author's mobile phone suddenly vibrated and opened it. It was a text message sent by Ali Yun. Click in to have a look, is a company in a project of the server CPU alarm message, the alarm content shocked!

  server CPU burst, and then received several text messages, the content of the text message is the same as the previous text message, all the servers in this project cluster CPU burst, this is not bad. Website usability alarm message also came, open the website and APP, can not open, has been in the "chrysanthemum", the author's mood is very sour, I do not know if you have experienced it.

Often many problems occur in an instant, making you feel very "surprised", so the psychological quality of the operation and maintenance personnel should be very strong, and they should be able to face all the stimuli calmly at any time.

What's wrong?

Log in to the server first. If the server CPU is full, the operation of logging in to the server will also be affected. Take a look at top first:

If you log in to other servers in the cluster, it will be the same:

This project is mainly based on php programs, so the servers in the cluster are not occupied by other processes except php-fpm processes. Note that the number of running processes is always high and a large number of processes are not released. If there is a problem with the tuning of PHP process parameters, let's take a look at the process configuration of php:

All of the company's servers are standard (CPU is 16 cores and 32 gigabytes of memory). On average, a php-fpm process takes up about 2m of memory, setting the maximum number of child processes to 800and starting processes to 100. if calculated in this way, the parameters are all within a reasonable range and there should be no problem.

Pm.max_children = 800 # php-fpm maximum number of child processes pm.start_servers = 100 # initial php-fpm processes in dynamic mode pm.min_spare_servers = 100 # minimum number of php-fpm processes in dynamic idle state pm.max_spare_servers = 200 # maximum number of php-fpm processes in dynamic idle state

Note: there are two ways for php-fpm process pool to open processes. One is static, which directly starts a specified number of php-fpm processes without increasing or decreasing them.

The other is dynamic, which starts with a certain number of php-fpm processes, dynamically increases the number of php-fpm processes to the upper limit when the number of requests increases, and automatically releases the number of idle processes to a lower limit when idle. These two different execution methods can be adjusted according to the actual needs of the server.

Ps 、

Iostat 、

Free 、

Iftop, and other ways to check the process, io, memory and bandwidth are not abnormal, nor received other alarms, ecs server, slb load traffic is not abnormal, strange?

First solve the problem, take a server to restart nginx, php services. No, it's the same after reboot. CPU is full instantly. Is it possible that php can't find it when it requests the redis service and is stuck there all the time?

Log in to redis to see that the usage of redis memory, cpu, and the number of connections is stable, and the connectivity between the server and redis is also Ok:

There is no activity at this time, ah, check the log quickly.

The problem comes from CC***.

Take a look at the nginx log. Error.log did not throw an exception log. Let's take a look at access.log:

Found a suspicious request, tail-f access.log tracked for a period of time, found that a lot of ip kept requesting the url "https:// company domain name / captcha/new.html?height=35&*=9999", why so many IP will keep requesting this url? Check and test, and find that this url is the verification code of the login page, as shown in the following figure. Users need a verification code to log in when they log in:

The author began to guess whether the CAPTCHA had been used. First, go to waf to check the traffic visits during this period:

From the data of waf, we can see that the business traffic suddenly increases, and we do not engage in activities or kill seconds. This will not happen in the normal traffic at this point. Let's take a look at the access to the full waf log and waf overview:

Let's take a look at the above picture, the author casually cut a page of the picture, each GET is from a different IP, probably counted, not less than thousands of IP, at this time some friends have thought of, limit these IP. If you do IP restrictions, thousands of IP to limit the head is not crazy, and you do not know which IP is the real user's request IP.

Let's take a look at other charts:

As can be seen from the above figure, in the full log of waf, it is also found that the log request is the same as that of nginx. In the number of visits, this url has been requested nearly 300000 times. Imagine who will click on your CAPTCHA all the time. From this it can be concluded that it has been cc***.

The contest between waf and cc

Now that it has been cc***, it must be dealt with. If you do not use waf, how can it be solved by relying solely on the server? Use nginx or iptables to limit the number of visits to a single ip, change the web port, or block ip, and so on. We can discuss it together. There are good suggestions and ways to learn and make progress together.

Now that the author uses waf here, let's take a look at how to play between waf and cc.

1. First, enter the custom cc configuration, as shown below:

2. Add a custom rule based on the URI found previously, as shown below:

It means that when a single IP accesses the destination address (when the prefix matches, that is, when the prefix matches to the prefix URI of / captcha), once the IP20 is accessed more than 3 times in 5 seconds, the IP20 minute is blocked.

Description:

URI: specify the specific address to be protected. For example, protect a user registration interface, / register. Input parameters such as / user?action=login are supported. Matching rules: exact match or prefix match.

An exact match means an exact match, and the request address must be exactly the same as the one configured here before it will be counted.

A prefix match contains a match, and will be counted whenever the requested URI starts with the configuration here (for example, / register.html will be counted). Detection duration: specifies the period of statistical visits. Need to match the number of visits. Number of single IP accesses: specifies the number of times a single source IP is allowed to access the URL during the statistical period. Blocking type: specify the action after the trigger condition, which can be blocked or man-machine identification.

Blocking: directly disconnect the connection after triggering the condition.

Man-machine identification: after triggering the condition, the client is redirected and verified before it is released. Blocking time: specifies the time to perform the blocking action.

3. After configuring the custom CC, let's see if the effect has worked. As shown in the following figure:

As can be seen from the yellow lines in the image above, the custom configuration of CC*** interception works, and the × × lines are not displayed in the time period before the intercept.

Now that CC has been stopped, is the business normal? Go back to the server to check, the server's CPU has indeed returned to normal, to see if the business is normal?

Open APP, website, visit the company's business has returned to normal.

After waiting for a period of time, there are still a small number of users' feedback can not be opened, what is the reason, analyze the waf:

A, in fact, the custom CC rule configuration on waf is very strict. Within 5 seconds, a single IP access will be blocked for more than 3 times. This strict rule will mistakenly kill a lot of real user requests. According to the company's business feedback, are there any normal users blocked? wait until CC*** is gone. You are relaxing the policy rules a little bit (for example, 5 seconds, single IP40 / 50 / 60, etc.), in a word, dynamically adjust the waf, do not adjust to death.

B, in addition, some companies export only one Ip address, and the clients have more than n clients and share one IP. In this case, the number of requests per second may be higher, which is also the request of normal users, and strict rules like the above may also be blocked.

CJWAF firewall, through man-machine identification, big data analysis, model analysis and other technical identification, to intercept. However, unlike interacting with programs, security is a confrontation between people, and the performance bottleneck of each website is also different. After finding an invalid, the website will be analyzed and directed. At this time, it needs to be dynamically adjusted according to the business situation to achieve a higher level of protection and protection effect.

D, especially the home page content, often requires OPS personnel and developers to analyze and determine which interface or URI is vulnerable to * (such as SMS API, CAPTCHA API, etc.), and protect it at the code layer and security level in advance.

Summary

  generally speaking, the protection of CC is not so simple, the means of camouflage is also thousands of changes, CC belongs to strong technical skills. Defense against CC can be done in a variety of ways, such as forbidding website agents to visit, making websites static pages as far as possible, limiting the number of connections, modifying the maximum timeout, and so on. In addition to using the above methods, it can also be prevented through a third-party firewall, which can also save a lot of worry.

This is the end of this chapter. If you like my article, please click "follow" at the top right corner!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.