Shulou(Shulou.com)06/01 Report--

This article mainly introduces how to analyze the Linux log, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

There is a lot of information in the log that you need to deal with, although sometimes it is not as easy as you might think. In this article we will introduce some examples of basic log analysis that you can do now (just search). We will also cover some more advanced analysis, but these will require your early efforts to make appropriate settings, which will save a lot of time later. Examples of advanced analysis of data include generating summary counts, filtering valid values, and so on.

We will first show you how to use several different tools on the command line, and then show how a log management tool can automate most of the heavy work to make log analysis easier.

Search with Grep

Searching for text is the most basic way to find information. The most common tool for searching text is grep. This command-line tool is available in most Linux distributions and allows you to search logs with regular expressions. A regular expression is a pattern written in a special language that can recognize matching text. The simplest pattern is to enclose the string you are looking for in quotation marks.

Regular expression

This is an example of looking for "user hoover" in the authentication log of a Ubuntu system:

$grep "user hoover" / var/log/auth.logAccepted password for hoover from 10.0.2.2 port 4792 ssh3pam_unix (sshd:session): session opened for user hoover by (uid=0) pam_unix (sshd:session): session closed for user hoover

It can be difficult to build accurate regular expressions. For example, if we want to search for a number like port "4792", it may also match timestamps, URL, and other unwanted data. The following example in Ubuntu matches an Apache log that we don't want.

$grep "4792" / var/log/auth.logAccepted password for hoover from 10.0.2.2 port 4792 ssh374.91.21.46-- [31/Mar/2015:19:44:32 + 0000] "GET / scripts/samples/search?q=4972 HTTP/1.0" 404,545 "-" surround search

Another useful tip is that you can use grep to do surround search. This will show you what the lines before or after a match are. It can help you debug things that cause errors or problems. Option B shows the first few lines, and option A shows the next few lines. For example, we know that when someone fails to log in as an administrator, their IP is not resolved in reverse, which means they may not have a valid domain name. This is very suspicious!

$grep-B 3-A 2 'Invalid user' / var/log/auth.logApr 28 17:06:20 ip-172-31-11-241 sshd [12545]: reverse mapping checking getaddrinfo for 216-19-2-8.commspeed.net [216.19.2.8] failed-POSSIBLE BREAK-IN ATTEMPTech Apr 28 17:06:20 ip-172-31-11-241 sshd [12545]: Received disconnect from 216.19.2.8: 11: Bye Bye [preauth] Apr 28 17:06:20 Ip-172-31-11-241 sshd [12547]: Invalid user admin from 216.19.2.8Apr 28 17:06:20 ip-172-31-11-241 sshd [12547]: input_userauth_request: invalid user admin [preauth] Apr 28 17:06:20 ip-172-31-11-241 sshd [12547]: Received disconnect from 216.19.2.8: 11: Bye Bye [preauth] Tail

You can also use grep with tail to get the last few lines of a file, or to track logs and print them in real time. This is useful when you make interactive changes, such as starting the server or testing code changes.

$tail-f / var/log/auth.log | grep 'Invalid user'Apr 30 19:49:48 ip-172-31-11-241 sshd [6512]: Invalid user ubnt from 219.140.64.136Apr 30 19:49:49 ip-172-31-11-241 sshd [6514]: Invalid user admin from 219.140.64.136

A detailed introduction to grep and regular expressions is beyond the scope of this guide, but Ryan's Tutorials has a more in-depth introduction.

Log management system has higher performance and stronger search ability. They usually index data and query in parallel, so you can quickly search GB or TB logs in seconds. Grep, by contrast, takes minutes and, in extreme cases, even hours. The log management system also uses a query language similar to Lucene, which provides a simpler syntax to retrieve numbers, fields, and others.

Parsing command line tools with Cut, AWK, and Grok

Linux provides several command-line tools for text parsing and analysis. It's useful when you want to parse a small amount of data quickly, but it can take a long time to process a large amount of data.

Cut

The cut command allows you to parse fields from delimited logs. A delimiter is an equal sign or comma that can separate fields or key-value pairs.

Suppose we want to resolve the user from the following log:

Pam_unix (su:auth): authentication failure; logname=hoover uid=1000 euid=0 tty=/dev/pts/0 ruser=hoover rhost= user=root

We can use the cut command to get the text of the eighth field split with an equal sign as follows. This is an example on a Ubuntu system:

$grep "authentication failure" / var/log/auth.log | cut-d'='- f 8roothooverrootnagiosnagiosAWK

In addition, you can also use awk, which provides more powerful parsing of fields. It provides a scripting language that you can filter out almost anything irrelevant.

For example, suppose we have the following line of log on the Ubuntu system, and we want to extract the user name for which the login failed:

Mar 24 08:28:18 ip-172-31-11-241 sshd [32701]: input_userauth_request: invalid user guest [preauth]

You can use the awk command as follows. First, a regular expression / sshd.*invalid user/ is used to match the sshd invalid user line. Then use {print $9} to print the ninth field based on the default delimiter space. This outputs the user name.

$awk'/ sshd.*invalid user/ {print $9}'/ var/log/auth.logguestadmininfotestubnt

You can read more about how to use regular expressions and output fields in the Awk user's Guide.

Log management system

Log management system makes parsing easier, so that users can quickly analyze a lot of log files. They can automatically parse standard log formats, such as common Linux logs and Web server logs. This saves a lot of time because you don't have to think about writing your own parsing logic when dealing with system problems.

The following is an example of an sshd log message that parses each remoteHost and user. This is a screenshot from Loggly, a cloud-based log management service.

You can also customize parsing for non-standard formats. A common tool is Grok, which uses a library of common regular expressions to parse raw text into structured JSON. Here is a case configuration of Grok parsing kernel log files in Logstash:

Filter {grok {match = > {"message" = > "% {CISCOTIMESTAMP:timestamp}% {HOST:host}% {WORD:program}% {NOTSPACE}% {NOTSPACE}% {NUMBER:duration}% {NOTSPACE}% {GREEDYDATA:kernel_logs}"}}

The following figure shows the output of parsed Grok:

Filter with Rsyslog and AWK

Filtering allows you to retrieve a specific field value instead of full-text search. This makes your log analysis more accurate because it ignores unnecessary matches from other parts of the log information. In order to search for a field value, you first need to parse the log or at least have a way to retrieve the event structure.

How to filter applications

Usually, you may only want to see the log of an application. It will be easy if your application saves all the records in one file. It can be complicated if you need to filter an application in an aggregate or centralized log. There are several ways to do this:

Use the rsyslog daemon to parse and filter the logs. The following example writes the log of the sshd application to a file called sshd-message, and then discards the event so that it does not repeat itself elsewhere. You can add it to your rsyslog.conf file to test this example.

: programname, isequal, "sshd" / var/log/sshd-messages

& ~

Use a command-line tool like awk to extract the value of a specific field, such as the sshd user name. The following is an example of a Ubuntu system.

$awk'/ sshd.*invalid user/ {print $9}'/ var/log/auth.log

Guest

Admin

Info

Test

Ubnt

Use the log management system to automatically parse the log, and then click filter on the desired application name. The following is a screenshot of extracting the syslog domain in the Loggly log management service. We filter the application name "sshd", as shown in the Venn diagram icon.

How to filter errors

A person most wants to see errors in the log. Unfortunately, the default syslog configuration does not directly output the severity of errors, making them difficult to filter.

Here are two ways to solve the problem. First, you can modify your rsyslog configuration to output the severity of the error in the log file to make it easy to view and retrieve. You can use pri-text to add a template in your rsyslog configuration, like this:

":% timegenerated%,%HOSTNAME%,%syslogtag%,%msg%n"

This example will be output in the following format. You can see the err that indicates the error in this message.

: Mar 11 18 pam_authenticate 18 00 Authentication failure hooverlue VirtualBox Magi su [5026]:

You can use awk or grep to retrieve error messages. In Ubuntu, for this example, we can use some grammatical features, such as. And >, they only match this domain.

$grep '.err >' / var/log/auth.log: Mar 11 18 pam_authenticate 1815 00 lens hooverrain VirtualBoxMagi su [5026]:, pam_authenticate: box

Your second option is to use a log management system. A good log management system can automatically parse syslog messages and extract error domains. They also allow you to filter specific errors in log messages with simple clicks.

The following is a screenshot from Loggly, showing the syslog domain of the highlighted error severity, indicating that we are filtering the error:

Thank you for reading this article carefully. I hope the article "how to analyze Linux logs" shared by the editor will be helpful to everyone. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.