Shulou(Shulou.com)06/01 Report--

This is a real case, which took place in January 2016, when the author made a complete record and published it on 51cto's blog entitled "recording a linux virus removal process." this article topped the 51cto weekly likes list in mid-January for many days. The general content is that the author's server is affected by *, implanted by * *, and then this * constantly launches ddos tools, which takes up a lot of bandwidth, which shows that the harm is still quite great. The article records the process of how the author clears this problem manually. I am not going to explain how to solve this problem here. Please refer to the source article for detailed solutions. Here, I would like to look at this problem on the other hand, that is, how to prevent this problem in advance and find this problem from time to time. If we can predict it in advance, or find this problem as soon as possible, we can greatly reduce a lot of losses.

A brief introduction to the process, which is not described in detail in the article, can only be guessed from personal experience. There is a sentence in the article is that I checked the WEB log, found nothing abnormal, check the database is normal, there is no error log, check the system log, did not see anything abnormal, but the system login log was cleared. It can be judged that * must have entered the system. This * has some experience and cannot be regarded as a novice, because he deletes the complete login log, so he can hide himself, such as login ip address and other information, which brings great challenges to traceability, and there may be no way to trace the source. The main reasons for this problem may be as follows: 1, weak password, may be scanned by others; 2, system loopholes, perhaps operating system vulnerabilities have not been repaired in time; 3, business logic vulnerabilities; 4, backdoor, it is also possible that the previous personnel deliberately left the backdoor on this server. But most of these problems can be found in the usual monitoring, such as log monitoring, performance monitoring, etc., so that the problems can be found in advance, during or even after the event. There are many of these tools, you can find their own, the following to the author is familiar with the SeciLog to analyze how to find the problem in the first place.

It is very necessary to deploy a centralized log analysis system in a system. If the local log is cleared as mentioned in the article, and the centralized log system can find the log information at that time, this is very helpful for troubleshooting and traceability forensics. What else can the centralized log collection system do besides collecting logs centrally? it can also analyze logs from time to time and give an alarm.

The first thing any behavior needs to do is to step on the spot, test, and step on the spot. It is easy to know that there is a living system here. Generally, it is easy to know through scanning, dns query, and so on. There is basically no way to avoid this. A large number of host scans occur on the Internet every day. When we know that there is a living machine, what to do next, that is, port scanning, weak password scanning, vulnerability scanning and other steps, now many machines are cloud services, and the hardware is no longer under their own control, so it is difficult to effectively prevent this. However, weak password scanning can be found and recorded, because since it is weak password scanning, there will be a large number of login activities, resulting in a large number of login failure logs. Let's first take a look at the login failure log under linux.

Jan 6 13:11:00 localhost sshd [3258]: Failed password for root from 192.168.21.1 port 53328 ssh3

From the log, we can see the login time, process number, login failure behavior, login account, source ip, login port, protocol and other information, we can see that these information is very rich. Let's take a look at what the log system analyzes.

As you can see from the above figure, the log analysis system deals with the main dimensions of login, including login source address, port, login account, protocol, time, type, and so on. This is much more convenient than looking directly at the original log. But that's not enough. we can't stare at the journal every day. Yes, so it is more convenient to generate an alarm when the system sends *. Here is the analysis of how the system generates an alarm and reduces false positives. Because the system is often logged in by people, people sometimes enter the wrong password, if the wrong behavior of each login will produce an alarm, it will produce a large number of false positives. How about that? it makes more sense. The author believes that it is more reasonable to have more login failures than the threshold in a short period of time. For example, if there are more than 10 login failures in the same ip in three minutes, it is considered to be a password guess. This kind of alarm is meaningful. Of course, everyone has their own judgment on this threshold. For example, I think there should be an alarm twice in a short period of time. Let's take a look at the alarm message.

As you can see from the figure above, when the system has a * behavior, the system will automatically generate an alarm. And this alarm can be sent by email, and if your email is bound to your phone, you can find the behavior in the first place.

This behavior has not been successful yet, but it is worth paying attention to. For example, IP can be blocked, and the system also supports automatic IP sealing, but it is better to do it manually just to be on the safe side. The above behavior is unsuccessful, but most of the time successful login behavior should be paid more attention to, but we can't all pay attention to these behaviors. The author thinks that we should focus on logging in during non-working hours and non-working places. This behavior is very dangerous. These SeciLog are also supported.

First, take a look at the definition:

The default non-working hours in the system are 0: 00 a.m. to 8: 00 a.m. and 20:00 to 24:00, the definition of non-working places is mainly a whitelist of IP addresses, so there is no screenshot.

Take a look at the log of successful login.

Jan 17 10:34:31 seciv sshd [2836]: Accepted password for tomcat from 172.206.160.155 port 38874 ssh3

Log time and login ip, which are also recorded in the log, can be analyzed by judging the two, non-working hours login and non-working place login.

It can be seen that the system is easy to get non-work place login, the same non-working hours are similar, there is no screenshot here.

Of course, these are not all of the monitoring. When the system logs in, operations such as deleting secure and modifying rsyslog.conf files occur, which is even more dangerous. The system also supports alarms for sensitive file operations.

It can be seen that when the user edits the rsyslog.conf file, the system will also generate an alarm.

The system also supports the monitoring of system performance, and alarms will be generated when a certain performance index of the system exceeds the threshold. The system supports the monitoring of cpu, memory, swap, hard disk and network card traffic.

Aiming at the content of "recording a linux virus clearance process", we analyze how to prevent and find the problem from time to time. If the author of this paper can deploy a set of centralized log audit system in advance and configure it. It is believed that the problem can be found and solved in time so that the impact of the problem can be minimized.

However, the author has a doubt that many people only go back to pay attention to safety when there is a problem. At this time, the problem has already happened and the loss has already occurred. I think it's a matter of consciousness. You can only learn slowly through a large number of cases. I hope this article will help you. Thank you for reading.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.