Shulou(Shulou.com)06/01 Report--

If the operation and maintenance staff is regarded as a doctor to see a patient, the log is the patient's statement to himself. In many cases, the doctor needs to find out whether the patient's condition is serious, what kind of medicine is needed, and what type of medicine is needed by describing the patient. Therefore, there is a saying in the ancients that prescribe the right medicine to the case. This disease is the description of the patient and the judgment of the doctor, and a lot of tests are added to the more serious illness. The description of the patient and the data on the test sheet are very important to the doctor when he sees the doctor. In the same way, logs play a similar role in OPS, but unfortunately, logs are seriously underestimated in many OPS until there is insufficient disk space to realize that a large log file has been deleted, which can save space.

The content of operation and maintenance

As can be seen from the above figure, there are still many points of concern in the operation and maintenance, and any point may cause problems in the operation and maintenance. Therefore, most of the operation and maintenance personnel play the role of firefighters in fighting the fire, where there is a problem.

Let's take a look at the commonly used monitoring system, the interface is very beautiful, there are many functions, but there is a question is that you will stare at this interface every day? I feel that the vast majority of people will not, and many people will focus on the outliers, that is, when there is a problem with the system, you tell me where there is a problem, and then I am analyzing and dealing with it according to the problem. Of course, when dealing with it, the system will be used.

So what does all this have to do with the log?

The log itself is worthless, and it will be valuable only when it is analyzed and used. The log contains a lot of useful information, including not only the operation and maintenance level, but also the business level and security level. In many cases, what operation and maintenance needs is a unified alarm platform, but the alarm is mostly based on automated analysis of logs and other conclusions, so logs are very important.

What is a journal?

To put it simply, a log is the information recorded by a computer system, equipment, software, and so on under certain circumstances. The specific content depends on the source of the log. For example, the Unix operating system logs user login and logout messages, the firewall logs ACL pass and reject messages, and the disk storage system generates log information when a failure occurs or if some systems believe that a failure will occur. There is a lot of information in the log that tells you why you need to generate the log and what has happened to the system. For example, Web servers typically log when someone visits a Web page to request resources (pictures, files, and so on). If the page visited by the user needs to be authenticated, the log message will contain the user name. This is an example of log data: you can use a user name to determine who has accessed a resource. Through the log, IT managers can understand the health of the system, security, and even the status of the operation.

What can a journal do?

In a complete information system, the log system is a very important functional component. It can record all the behavior generated by the system and express it according to a certain specification. We can use the information recorded by the log system to debug the system, optimize the performance of the system, or adjust the behavior of the system according to this information. In the field of security, logs can reflect a lot of security behaviors, such as login errors, abnormal access and so on. Logs can also tell you a lot about events that occur on the network, including performance information, fault detection, and * detection. Logs can be a good source of forensics information to find out what happened after an accident. Logs can be used as audit trails for audits.

Start with a journal.

111.88.155.166-[17/Dec/2015:13:06:05 + 0800] "POST / login HTTP/1.1" 3020 "http://secilog.abc.com/login?langType=zh"" Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 "

This is a very common log recorded in nginx, and the details of the log can be found in the relevant documentation. Here is a brief explanation of the main content. From the log, you can get the visitor's IP, access time, time zone, request method, request page, return status, source, and so on. If you look closely at the requested page / login, you can guess that it is just a login request page. The important meaning of this log is login success.

From how this log corresponds to the indicators we are concerned about, let's go on to analyze.

The number of active users. Active users generally refer to how many regular users have logged in to the system on the same day. At this time, you will find that if you put the login log into the statistics of one day, you can know how many times you have successfully logged in in a day, but careful users can find that it is not accurate, because users can log in repeatedly, which will lead to repetition. That's right. Let's refine it. Let's analyze the number of successful logins without repeating the number of ip in a day. Whether it is closer to the real result, I feel that it can already explain the problem in terms of magnitude and trend.

There is no standard term for brushing users. My understanding is that the same person registers a large number of accounts for a certain purpose, and then carries out certain operations such as brushing. This kind of behavior is hard to put an end to 100%, but some interesting findings can be drawn from this log. If the same ip login is successful too many times a day, for example, a hundred times a day, each time interval is the same, indicating that this person is suspected of brushing, you can first find out and then further analysis.

The number of new users means how many users have successfully registered in a day. At this time, it can be compared to the login log. As long as you change the url of the login log to the url of the registration log, you can find out the number of new users in a day.

In the same way, the number of malicious registered users is similar, and there are many successful times for the same ip*** book in one day. The possibility of malicious registration of this ip is very high. Of course, further analysis is needed, such as whether ip is an exit ip in a building, and what the user does after registering.

From the above analysis, we can see a lot of content in the operation, such as the ranking of browsing products, user access time, user sources and so on.

Let's also analyze the safe behavior from this log:

111.88.155.166-[17/Dec/2015:13:06:05 + 0800] "POST / login HTTP/1.1" 2000 "http://secilog.abc.com/login?langType=zh"" Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 "

This is also a login log, and the only difference from the login log above is the server return value. One is 302 and the other is 200. What's the difference? 302 means that the server has made a page jump, 200 or returned to this page, from which it can be understood that this is a record of login failure. Well, with this record, you can find a lot of safe behavior.

Malicious password guessing, it is understandable that during the same period of time, a large number of login failures of users returned a lot of login failure records. We can find the rule from this definition, we enlarge the time to 5 minutes, when the same ip has more than 20 login failures in 5 minutes, we can basically conclude that we are doing password guessing. When password guessing is automatic and manual, what's the difference? Let's take a look at the content "http://secilog.abc.com/login?langType=zh", which means that the source of the post submission is" http://secilog.abc.com/login?langType=zh", which is initiated from this page. If this address is wrong, it is very likely to use tools for violent cracking.

By the same token, cc*** is easier to understand. The same ip accesses a large number of requests in a very short period of time, which can basically be thought of as cc***. Other webshell,sql injections can also be analyzed from the log, but it is not very accurate, because the log refers to the parameters of the get request, and the post parameters are not recorded normally.

From the above analysis, we can see that there are still a lot of valuable things in the log, but we haven't found it.

How to analyze the log

General log analysis mainly includes the following levels, the first is to collect logs, then format analysis of logs, and then filter or merge, and then alarm analysis of logs, and then storage.

Collection is mainly to support various protocols, such as syslog,sftp and so on.

Formatting analysis is the key point, after all, the format of each log is different. For example: the following figure shows a log of pix Firewall and ids, which analyzes meaningful dimensions from the messy and disordered content of the original log. Through these dimensions, we get a lot of valuable information, such as operating system, protocol and so on.

Log analysis

Log analysis related to keyword analysis, statistical analysis and correlation analysis.

Keyword analysis is to analyze the keywords in the log.

Statistical analysis is based on a certain law over a period of time.

Association analysis is used to find out the relationship between heterogeneous and heterogeneous event information in a large amount of audit information, and to formulate a reasonable audit strategy for the context in which there is related relationship information. through the combination of multiple heterogeneous events to judge the nature of the operation, explore the hidden relevance, and find the possible violations.

These things themselves are very complicated, and it's a lot of work if they have to be done from scratch. Of course, there are many good products on the market to support this function. For example, HP ArcSight,IBM Security QRadar SIEM and so on, but these products are very expensive products, is there any products that can be used for free? Yes: for example, foreign elk,ossim. Domestic SeciLog. Each of these products has its own advantages and disadvantages, it is up to everyone to choose. Elk is a semi-finished product, and it takes a lot of work to use it. Ossim is relatively a finished product, but the Sinicization of these two foreign products is not too good. The habits of these two foreign products are not very good for domestic users. Secilog is relatively balanced. Secilog is characterized by supporting syslog, snmp, jdbc, ftp/sftp and other protocols to collect or collect logs. Analyze the logs, format the logs, generate alarms, and store the full-text search index of the original logs and the formatted logs, support the collection of horizontal expansion clusters, and support the analysis and query of massive logs. Can analyze linux logs, windows logs, firewall logs, ids logs, business logs and other logs, support all text types of log storage and query, built-in 16 kinds of alarms: password guessing * *, non-working hours login, non-working place login, account guessing * *, password guessing * * successful, sensitive file operations, high-risk command operations, host scanning, port scanning, illegal outreach, sql injection Xss***, illegal access, sensitive file access, WebShell***,Cc***. Through the setting of alarm rules, it is easy to add other alarms. At the same time, the system can support business alarm, abnormal interface request, malicious brushing, large single alarm and so on through alarm rule configuration.

I'm glad you finished it. I hope it works for you.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.