Shulou(Shulou.com)06/03 Report--

This article mainly explains "what is the operation behavior of Linux system". Interested friends may wish to have a look at it. The method introduced in this paper is simple, fast and practical. Next, let the editor take you to learn "what is the operating behavior of the Linux system?"

In many cases, we may record and analyze the operation behavior of the host system for security audit or fault tracking. For example, adding a new user in the system, changing a file name, or executing some commands and so on, the more detailed the record in theory, the more conducive to the purpose of audit and troubleshooting. However, excess records can also bring a lot of trouble to the analysis, especially when the recording behavior of many hosts is sent to a fixed remote host. The more data, the greater the cost of analysis.

In fact, the vast majority of system behaviors are redundant, such as cron task planning, programs we trust, etc., which produce a large number of records, but are rarely used for audit analysis. Based on this requirement, when auditing system operations, we should at least add some filtering rules to avoid recording too much useless information, such as repetitive cron task operations, as well as sensitive information, such as command line operations with passwords. After meeting these requirements, we should follow the following guidelines when auditing the operation of the system:

Ignore records generated by cron,daemon

Ignore sensitive command line or script action records with passwords

Ignore records generated by monitoring users, such as nagios,zabbix,promethus, etc.

Ignore the operation of frequently generating logs

The second point is optional, and we recommend ignoring records when transferring to the remote log server in clear text. The fourth point needs to be emphasized. For example, if we record all the connect,accept network system call operations in a web host, although we can analyze all the network access requests of the host to achieve the purpose of security or fault location, these two system calls may generate a large number of logs in a short period of time, which will exert great pressure on kernel and network log transmission. This kind of audit like looking for a needle in a haystack is not recommended to be used directly on the mainframe online. it is recommended to enable it only when you need to locate the problem.

Below, we mainly introduce several ways to audit system operations:

History recording mode

Custom bash recording method

Snoopy recording mode

Auditd recording mode

EBPF recording mode

History recording mode

The history method is very traditional and simple. In essence, it sends historical commands to the syslog log, which can be used to simply record the user's command operation history. However, this approach has several important shortcomings and is not suitable for the purpose of audit:

Easily modified and bypassed

Recording is too simple to have contextual information (such as pid, uid, sid, etc.)

Unable to record actions within the shell script

Unable to record non-logged in actions

It is difficult to implement filtering rules

Custom bash recording method

Customized bash is unpopular, which essentially adds audit log to bash source programs. Developers can add some context information of operation commands based on this, but it is difficult to record the information of child processes. Its disadvantages are similar to the above history method:

Easily bypassed, users can use csh,zsh, etc.

Unable to record actions within the shell script

Filtering rules may be single

You may need to constantly update the bash version, which is a heavy workload, otherwise it is easy to be replaced by the distribution.

Snoopy recording mode

Snoopy mode is relatively novel, which essentially encapsulates execv,execve system calls and records all command operations in the way of system preloading (preload). For more information, you can refer to the previous article, snoopy, on how to record commands executed by the system. At present, most systems execute commands through execv,execve system calls, which has nothing to do with the session. In almost all cases, as long as commands are executed through these two system calls, the operation behavior will be recorded. From the latest version (2.4.8), snoopy has several advantages:

It is difficult to bypass. As long as PRELOAD is set, it will definitely be recorded.

Regardless of whether there is a tty session or not, execv,execve-related command line operations are logged, including detailed process context information

You can record the actions within the shell script, and most of the command line operations in the script will call execv,execve

You can record the parameters of the operation, such as specifying a user name, password, etc.

Filter rules are rich. You can ignore the specified daemon,uid or record only the specified uid.

The following log example:

Oct 27 11:34:31 cz-t1 snoopy [24814]: [time_ms:778 login:cz uid:0 pid:24814 ppid:24676 sid:24579 tty:/dev/pts/0 cwd:/root filename:/bin/uptime username:root]: uptime-p

The above log shows that root users have executed the uptime command, and the parameters contain all the process context information corresponding to-p. However, the disadvantages of snoopy are also obvious, including the following points:

Only operations related to execv,execve system calls are supported

If the rules are not set, there may be too many logs, which will cause a great burden on the log collection system.

Rules for filtering sensitive information are not supported.

In practical use, snoopy recording method can record all the command operation information in detail, which helps us to locate many difficult problems. However, we also need to avoid generating too much information by filtering rules. Snoopy filtering rules can meet the following requirements:

Ignore records generated by cron,daemon

Ignore records generated by monitoring users, such as nagios,zabbix,promethus, etc.

For example, the following configuration allows you to ignore the crond,my-daemon daemon and the zabbix user:

# zabbix uid is 992 filter_chain = exclude_uid:992;exclude_spawns_of:crond,my-daemon

Note: the filtering rules are implemented in the (filtering.c-snoopy_filtering_check_chain) function and are called by the log.c-snoopy_log_syscall_exec function. The filtering rules are ex post actions, that is, judging whether the filtering rules are satisfied when printing logs is not an ex ante behavior.

In addition, we have added exclude_comm filtering rules to snoopy, so we can ignore the commands specified by records, such as the following:

Filter_chain = exclude_uid:992;exclude_comm:mysql,mongo,redis-cli

Exclude_comm specifies to ignore commands executed by mysql,mongo and redis-cli tools, and many administrators or scripts often add user password information when using these tools, which is a dangerous behavior in a plaintext environment. Exclude_comm rules simply avoid the hidden danger of leaking sensitive information from common tools.

Auditd recording mode

Auditd recording mode itself has kernel level (kauditd process) support, it implements a large and comprehensive framework, which can monitor almost all the indicators you want to monitor, whether according to the access mode, system call or event type trigger, can meet the monitoring needs. Because it provides kernel-level support, it is essentially more powerful and robust than snoopy (which encapsulates only execv,execve system calls).

The generated log is also easy to view, and the context information and parameter information of the process are comprehensive, as shown below:

Type=SYSCALL msg=audit (1603800704.305 type=SYSCALL msg=audit 5304075): arch=c000003e syscall=59 success=yes exit=0 a0=1c79fd0 a1=1bf51a0 a2=1bd4450 a3=7ffe7270d320 items=2 ppid=95264 pid=99702 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=571973 comm= "mysql" exe= "/ usr/bin/mysql" key= "command" type=EXECVE msg=audit (1603800704.305 type=SYSCALL msg=audit 5304075): argc=5 a0 = "/ usr/bin/mysql" A1 = "- h" a2 = "127.0.0.1" a3 = "- P" a4 = "3301"

Auditd is a separate architecture as a whole. Auditctl can control the policy for kauditd to generate records, record events generated by kauditd will be sent to auditd daemons, and audisp can consume auditd records to other places. Its main tools include the following:

The policy rules of auditd are mainly set according to the-an or-w parameters, which can be saved to the default / etc/audit/rules.d/audit.rules configuration or dynamically adjusted by auditctl. It is worth noting that the loading of policy rules takes effect sequentially. When configuring exceptions, we need to add the exceptions to the appropriate location. For example, refer to the example given in auditd-best-practice. If you need to ignore command tools such as mysql,mongo, you need to add the following policy to the appropriate location (before the-an always,exit rule):

# ignore common tools-a never,exit-F arch=b64-F exe=/usr/bin/redis-cli-a never,exit-F arch=b64-F exe=/usr/bin/mysql-a never,exit-F arch=b64-F exe=/usr/bin/mongo.... # # Kernel module loading and unloading-an always,exit-F perm=x-F auidophile 1-F path=/sbin/insmod-k modules....

The-F filter fields that never and always can support are different. If you want to ignore the specified tool path according to exe, you can only implement it through never. Exe is the path of the execution tool, and you need to set its absolute value, which is not as convenient as snoopy's exclude_comm.

For more rule settings, see: audit-define

EBPF recording mode

EBPF is implemented in a newer version of the Linux kernel and provides a dynamic tracking mechanism. You can read the previous article Linux system dynamic tracking technology introduction to learn more about dynamic tracking. Bpftrace and bcc are tools based on eBPF mechanism to facilitate debugging and troubleshooting of the system. Bcc provides many toolsets, from application to kernel, all kinds of tools are available, such as execsnoop to record the execution of all execv,execve-related commands in the system:

#. / execsnoop PCOMM PID PPID RET ARGS bash 32647 32302 0 / bin/bash id 32649 32648 0 / usr/bin/id-un hostname 32651 32650 0 / usr/bin/hostname uptime 410 32744 0 / bin/uptime

For more detailed records, please refer to the bcc tool description. It is worth noting that eBPF is only applicable to Linux 4.1 +. According to the progress of eBPF development, eBPF's support after kernel-4.10 is relatively comprehensive. Try to choose a higher kernel version when using it online (such as Centos 8 and Debian 10, etc.). In addition, Readhat/Centos 7 supports eBPF features from version 7.6 (3.10.0-940.el7.x86_64), but the kernel version is lower and does not support all the features. Its main purpose is to try out this technology:

At this point, I believe you have a deeper understanding of "what is the operating behavior of the Linux system?" you might as well come to the actual operation! Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.