Shulou(Shulou.com)06/01 Report--

Background

There is a distributed service with four machines, no more, no less, it is troublesome to view logs on each machine, and it seems overqualified to use distributed log management systems such as Flume,Logstash, ElasticSearch, Kibana, etc., so I think of rsyslog that comes with centos.

Brief introduction

Rsyslog can be simply understood as a superset of syslog. In the old version of Linux system, Red Hat Enterprise Linux defaulted to syslog as the logging tool for the system in 3-4-5, and Rsyslog was used by default since RHEL 6.

Rsyslog is the program responsible for collecting syslog and can be used instead of syslogd or syslog-ng. Among these syslog processors, I personally think that rsyslog is the most powerful. Its characteristics include:

Support for exporting logs to various databases, such as MySQL,PostgreSQL,MongoDB,ElasticSearch, etc.

Reliable data transmission through RELP + TCP (based on this combined with rich filtering conditions, a reliable data transmission channel can be established for other applications to use)

Fine output format control and powerful message filtering ability

High-precision timest queue operation (memory, disk and mixed mode, etc.); support for data encryption and compression transmission.

Version view $rsyslogd-versionrsyslogd 3.22.1, compiled with: FEATURE_REGEXP: Yes FEATURE_LARGEFILE: Yes FEATURE_NETZIP (message compression): Yes GSSAPI Kerberos 5 support: Yes FEATURE_DEBUG (debug build, slow code): No Atomic operations supported: Yes Runtime Instrumentation (slow code): NoSee http://www.rsyslog.com for more information. Install yum-y rsyslog# to see if rsyslogrpm-qa is installed | if grep rsyslog# needs other components (mysql module, log rotation) yum-y rsyslog-mysql yum-y logrotate start / stop / etc/init.d/rsyslog start/etc/init.d/rsyslog stop/etc/init.d/rsyslog restart// help documentation man rsyslogd Or enter an incorrect command $rsyslogd-- helprsyslogd: invalid option -'- 'usage: rsyslogd [- c] [- 46AdnqQvwx] [- l] [- s] [- f] [- I] [- N] [- M] [- u] To run rsyslogd in native mode, use "rsyslogd-c3" For further information see http://www.rsyslog.com/doc configuration

Rsyslog configuration files can be written in a variety of ways:

Sysklogd (some structures are not compatible with new features)

Legacy rsyslog (words that begin with "\ $", e. G. $ModLoad imtcp.so)

RainerScript (a new format that is the most recommended, especially when complex configurations are required).

The configuration in this paper is relatively simple, so we use the configuration writing method of legacy rsyslog. For more details, please see http://www.rsyslog.com/doc/master/configuration/basic_structure.html#statement-types

Simple instance of configuration file

Here is an example:

$less / etc/rsyslog.conf # rsyslog v3 config file# if you experience problems Check# http://www.rsyslog.com/troubleshoot for assistance#### MODULES # $ModLoad imuxsock.so # provides support for local system logging (e.g. Via logger command) $ModLoad imklog.so # provides kernel logging support (previously done by rklogd) # $ModLoad immark.so # provides-- MARK-- message capability# Provides UDP syslog reception#$ModLoad imudp.so#$UDPServerRun 51customers Provides TCP syslog reception#$ModLoad imtcp.so # $InputTCPServerRun 514 clients # GLOBAL DIRECTIVES # Use default Timestamp format$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat# File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit#$ActionFileEnableSync on#### RULES # Log all kernel messages to the console.# Logging much else clutters up the screen.#kern.* / dev/console# Log anything (except mail) of level info or higher.# Don't log private authentication messageshandling mail.noneten authpriv.none Cron.none / var/log/messages# The authpriv file has restricted access.authpriv.* / var/log/secure# Log all the mail messages in one place.mail.*-/ var/log/maillog# Log cron stuffcron.* / var/log/cron# Everybody gets emergency messages*.emerg * # Save news errors of level crit and higher in a special file.uucp News.crit / var/log/spooler# Save boot messages also to boot.loglocal7.* / var/log/boot.log profile module

The configuration file looks at less / etc/rsyslog.conf. The configuration of Rsyslog mainly includes the following modules:

Modules, module, configuration loaded module, such as: ModLoad imudp.so configuration loading UDP transport module

Global directives, global configuration, configure the global properties of the ryslog daemon, such as the main message queue size (MainMessageQueueSize)

Rules, rule (selector + action), each rule line consists of two parts, the selector part and the action part, which are separated by one or more spaces or tab, the selector part specifies the source and log level, and the action part specifies the corresponding action

Template (templates)

Output (outputs)

Commonly used modules

Imudp, traditional UDP transmission, lossy

Imtcp, based on TCP plaintext transmission, loses information only in certain circumstances and is widely used

Imrelp,RELP transmission, no loss of information, but only available in rsyslogd 3.15.0 and above

More referenc

Rule (rules) rule selector (selectors)

Selector also consists of two parts, facility and priority, by the dot. Separate. The first part is the message source or log facility, and the second part is the log level. Multiple selectors are used to separate, such as: * .info; mail.none.

Log facilities are as follows:

Auth (security), authpriv: authorization and security-related messages

Kern: messages from the Linux kernel

Mail: messages generated by the mail subsystem

Cron: information about the cron daemon

Daemon: information generated by the daemon

News: network message subsystem

Lpr: print related log information

User: information about the user's process

Local0 to local7: reserved, used locally

Log levels are (ascending):

Debug: information that contains detailed development intelligence, usually used only when debugging a program.

Info: intelligence information, normal system messages, such as harassment reports, bandwidth data, etc., do not need to be processed.

Notice: it's not an error condition, and it doesn't need to be dealt with immediately.

Warning: warning message, not an error, for example, 85% of the system disk is used.

Err: error, it is not very urgent, it can be fixed within a certain period of time.

Crit: important situations, such as hard drive error, loss of backup connection.

Alert: problems that should be corrected immediately, such as corruption of the system database and loss of ISP connections.

Emerg: in an emergency, the technician needs to be notified immediately.

Configuration of logging facilities:

. It means that messages higher than the following will be recorded.

. = means that only the latter message level will be recorded.

.! Except for the following message level, everything else will be recorded. I don't know why I failed in rsyslogd 4.6.2. no, no, no.

Can be used for multiple selectors; separate.

Local0.=debug / home/admin/applogs/app-name/debug.loglocal0.err;local0.warning;local0.info / home/admin/applogs/app-name/info.loglocal0.err / home/admin/applogs/app-name/error.log Action (action)

Action is part of the rule description, after the selector, and the rule is used to process messages. In general, the message content is written to a log file, but other actions can also be performed, such as writing to a database table or forwarding to another host.

In the previous example, it is written to the local file:

# The authpriv file has restricted access.authpriv.* / var/log/secure

It can also be written to the mysql database

# modules. To write logs to mysql, you need to load the ommysql module $ModLoad ommysql # rule, send to mysql#*.*: ommysql:database-server,database-name,database-userid,database-password*.*: ommysql:127.0.0.1,Syslog,syslogwriter,topsecret

For more information about configuring the capacity to send messages to the database, please refer to: http://www.rsyslog.com/doc/master/tutorials/database.html

Configuration of action:

Save to a file, cron.*-/ var/log/cron.log if there is a-before the path, it means that each time the log is output, the specified log file is not synchronized (fsync). The file path can be either static or dynamic. Dynamic files are added before the template. Definition.

The format of the log sent over the network is as follows: @ [()]: [] @ indicates the use of UDP protocol. @ @ indicates the use of the TCP protocol. You can use: Z for zlib compression and NUMBER for compression level. Multiple options are used to separate. For example:. @ 192.168.0.1 # use UDP to send logs to 192.168.0.1 *. * @ @ example.com:18 # to port 18 of "example.com" using TCP. @ (Z9) [2001 VRV 1] # use UDP to send messages to 2001 VG 1, enabling zlib 9-level compression

Cron.* ~ discard all information, that is, the log will not be seen by actions after the configuration. Depending on the version of rsyslog, change ~ to stop if you have the following warning message.

Template (templates)

Templates allow you to specify the format of log information, can also be used to generate dynamic file names, or can be used in rules. The definition is as follows, where TEMPLATE_NAME is the name of the template and PROPERTY is some of the attribute parameters supported by rsyslog itself.

$template TEMPLATE_NAME, "text% PROPERTY% more text", [OPTION]

Examples of use:

$template DynamicFile, "/ var/log/test_logs/%timegenerated%-test.log" $template DailyPerHostLogs, "/ var/log/syslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages.log" * .info? DailyPerHostLogs*.*? DynamicFile

The properties we use in the template can be explained by referring to the official documentation. The timegenerated used in the example refers to the timestamp when the message is received.

Output (outputs)

Output channels provide protection for the types of output that users may want, which should be defined before using them in rules. The definition is as follows, where NAME specifies the name of the output channel, FILE_NAME specifies the output file, MAX_SIZE specifies the size of the log file in bytes, and ACTION specifies the operation when the log file arrives at MAX_SIZE.

$outchannel NAME, FILE_NAME, MAX_SIZE, ACTION

Use the output channel in the rule in the following format:

Selectors: omfile:$NAME

Example:

$outchannel log_rotation, / var/log/test_log.log, 104857600, / home/joe/log_rotation_script*.*: verification of omfile:$log_rotation configuration

You can verify that the configuration file is configured correctly with the following command:

Sudo rsyslogd-f / etc/rsyslog.conf-N4

The value after-N represents the version specified after-c when rsyslog starts.

Log information can be sent manually with the following command:

Logger-p local0.info "hello world" log file Rotating

As log files get larger and larger, this will not only cause performance problems, but also tricky to manage logs. When a log file is rotated, a new log file is created and the old log file is renamed. These files are retained for a period of time, and once a certain number of old logs are generated, the system will delete some of the old logs.

Logrotate profile instance

Logrotate is called through the cron task and is automatically created during installation, so the logrotate cannot be seen through the ps command, so you can see the scheduled task call: cat / etc/cron.daily/logrotate:

#! / bin/sh/usr/sbin/logrotate / etc/logrotate.conf > / dev/null 2 > & 1EXITVALUE=$?if [$EXITVALUE! = 0]; then / usr/bin/logger-t logrotate "ALERT exited abnormally with [$EXITVALUE]" fiexit 0

File execution under cron.daily is configured through / etc/crontab:

$cat / etc/crontabSHELL=/bin/bashPATH=/sbin:/bin:/usr/sbin:/usr/binMAILTO=rootHOME=/# For details see man 4 crontabs# Example of job definition:#.-minute (0-59) # |.-hour (0-23) # | |.-day of month (1-31) # |.-month (1-12) OR jan Feb,mar,apr... # |.-day of week (0-6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat# | | # * user-name command to be executed0 0 * root run-parts / etc/cron.daily # regular execution of cron.daily

The configuration file for logrotate is / etc/logrotate.conf. Here is an example:

# see "man logrotate" for details# rotate log files weeklyweekly# keep 4 weeks worth of backlogsrotate create new (empty) log files after rotating old onescreate# uncomment this if you want your log files compressed#compress# packages drop log rotation information into this directoryinclude / etc/logrotate.d# no packages own wtmp Or btmp-- we'll rotate them here/var/log/syslog {rotate 7 daily missingok notifempty delaycompress compress postrotate invoke-rc.d rsyslog reload > / dev/null endscript} / var/log/cron.log/var/log/debug/var/log/messages {rotate 4 weekly missingok notifempty compress delaycompress sharedscripts postrotate invoke-rc.d rsyslog reload > / dev/null endscript} # system-specific logs may be configured here

Syslog's log files are rotated every day, keeping 7 old logs. Other log files are rotate once a week and keep 4 old logs.

Logrotate configuration item

We can get all the parameters and detailed descriptions through man logrotate. Some of these are listed here:

Daily specifies that the dump cycle is daily

Weekly specifies that the dump cycle is weekly

Monthly specifies that the dump cycle is monthly

Compress dumps logs through gzip compression

Use this parameter when nocompress does not need compression

Copytruncate is used for opening log files to back up and truncate the current log

Nocopytruncate backs up log files but does not truncate

Missingok if the file does not exist, proceed to the next file without reporting an exception

Nomissingok if the file does not exist, an exception is reported (default configuration)

Create mode (file permissions) owner (owner) group (group) dump files and create new log files using the specified file mode

Nocreate does not create a new log file

When delaycompress and compress are used together, the dumped log files are not compressed until the next dump

Nodelaycompress overrides the delaycompress option and the dump is compressed at the same time.

The error message from the errors address dump is sent to the specified Email address

Ifempty dumps even empty files, (the default option for logrotate)

Notifempty does not dump if it is an empty file

Mail address sends the dumped log file to the specified E-mail address

Log files are not sent when nomail dumps

The log files after olddir directory dump are placed in the specified directory and must be on the same file system as the current log files.

The log file after noolddir dump is placed in the same directory as the current log file.

Commands that prerotate/endscript needs to execute before the dump can be put into this pair, and the two keywords must be on separate lines

The commands that postrotate/endscript needs to execute after the dump can be put into this pair, and the two keywords must be on separate lines.

Rotate count specifies the number of times to dump the log file before it is deleted. 0: no backup, 5: keep 5 backups.

Tabootext [+] LIST tells logrotate not to dump files with the specified extension, which default to: .rpm-orig, .rpmsave, v, and ~

Size SIZE log files are not dumped until they reach the specified size. Size can specify bytes (default) and KB (sizek) or MB (sizem).

Example

Sudo vim / etc/rsyslog.conf

# Provides UDP syslog reception$ModLoad imudp.so$UDPServerRun 514$ template ipAndMsg, "[% fromhost-ip%]% $now%%msg%\ n" local0.=debug / home/admin/applogs/app-name/debug.log;ipAndMsglocal0.err;local0.warning;local0.info / home/admin/applogs/app-name/info.log;ipAndMsglocal0.err / home/admin/applogs/app-name/error.log;ipAndMsg

Sudo service rsyslog restart

Sudo service syslog/syslog-ng stop

Sudo vim / etc/logrotate.conf

/ home/admin/applogs/app-name/debug.log/home/admin/applogs/app-name/info.log/home/admin/applogs/app-name/error.log {daily create 0664 root root rotate 30 missingok nocompress notifempty dateext postrotate / etc/init.d/rsyslog restart > / dev/null 2 > & 1 endscript}

Note that finally, it must be added:

Postrotate / etc/init.d/rsyslog restart > / dev/null 2 > & 1endscript

Because after logrotate, even if it has been removed, rsyslog still holds this file operation handle and will continue to write to the original file (the file by rotate), even if it has been renamed, so you need restart rsyslog to reopen the newly created file with the same name under logrotate.

There is another way to avoid rebooting, but some data will be lost. Logrotate provides copytruncate. The default instruction create practice is to move the old file, create a new file, and then use the script to reopen the new file; while copytruncate is used to copy and then empty, first copy an old log, and then treat the original file, the original file handle of the whole process has not changed, so there is no need for reopen, the service can be uninterrupted, but this process will lead to some data loss.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.