Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to deal with Linux emergency response". The content of the article is simple and clear, and it is easy to learn and understand. Please follow the editor's train of thought to study and learn how to deal with Linux emergency response.

Background

The customer's monitoring system found abnormal behavior, I temporarily replace the emergency colleagues to deal with it.

To connect to the server, first check the abnormal process information through ps auxef and netstat-tulnp commands, and sure enough, two abnormal processes xmp and [atd] are found.

View the program locations of these two processes through ls-al / proc/ [pid] / exe, where [pid] is the process id of xmp and [atd] processes.

Finally, confirm that xmp is in the / lib/PROXY/ directory, where there are two files, one is xmp, the other is config.json [atd] in the / var/spool/at/.sqe/ directory, there are many files in this directory, including [atd], cyc.acc, seed, stealth, randfiles, etc.

Upload the two processes to virustotal, and more than half of the antivirus software reports virus.

Execute stat / lib/PROXY/xmp, stat / var/spool/at/.sqe/ [atd], and find that the Change time of these two files are both in 23 and 24.

So it is suspected that it was invaded around the 23rd, check history, / var/log/secure found that the files have been emptied, check / root/.ssh/known_hosts found more than 600 records. I can't find any clues, so I can only think that ssh broke the login.

After restarting the server, I found that the [atd] process still exists and should have joined the boot. I used a relatively rough way to locate the boot and execute it in the root directory.

Grep-rn'\ [atd\]'*

Huangtian lived up to his painstaking efforts, and sure enough, I found it. There is code to start [atd] in / bin/seed. This script is very simple, just cd to / var/spool/at/.sqe/ and then execute [atd].

Next, I go to the / etc directory and continue to execute grep-rn seed *. The result of this command is many lines. After filtering one by one, I find that there is a new command seed on one line of / etc/rc.sysinit, which can explain why [atd] can boot, but the boot entry of xmp is not found, and xmp will not start itself with the server restart.

Look at the process name of [atd] and guess that this is a process that executes scheduled tasks. This process listens on the udp port and guesses that the attacker should control the server through this process and execute commands, including starting xmp.

Looking back at xmp, we can see from the config.json file that it is a Monroe coin mining virus.

"pools": {"algo": null, "coin": "monero", "url": "pool.supportxmr.com:80", "user": "44wuEu1F6UMDzAu2ByHjKGRR4WiU33zJW6bdHPrHaHbLWYHTyqJUiqG47yvaJof8gfd1HbMR1WhmsDJcX7yhVx8bU8PHRtBx", "pass": "HERCULE", "rig-id": null, "keepalive": true, "enabled": true "tls": false, "tls-fingerprint": null, "daemon": false}]

The final cleanup process is simple: delete the line / etc/rc.sysinit seed, delete / bin/seed, delete / lib/PROXY, delete / var/spool/at/.sqe/

The hardening method is to configure some unnecessary ports with iptables to reject all connection requests and change the ssh password to an uncommon strong password.

Emergency response process

Back to the point, what should be the standard process for emergency response? Security+ gives a set of processes:

Preparation-> Identification-> Containment-> Eradication-> Recovery-> Lessons learned

With the example in the background above, Preparation is the channel through which the front-line staff provides me with access to the server. Identification is when I found xmp and [atd] confirmed that the server was infected with a virus. Containment quarantines all systems that may be affected, including more than 600 hosts found by known_hosts above. Eradication clears all affected hosts based on the above cleanup. After the Recovery is cleared, the quarantine is released and the business system is restored. Lessons learned summarizes and reflects on events, on the one hand, to reduce the discovery of security incidents from the source, on the other hand, to improve the efficiency of emergency response.

The above emergency response is still very one-sided. I have collected a series of emergency response experiences shared by netizens and sorted them into chapters for later reference.

I divided the emergency response process into three parts, namely, [1] intrusion site, [2] attack maintenance, and [3] intrusion reasons. I will start from these three aspects.

Invade the scene

The so-called intrusion site refers to the site environment where the server is suspected of poisoning. Generally speaking, the server is suspected to be poisoned with abnormal phenomena, such as abnormal network traffic, abnormal ports, abnormal cpu/ memory utilization and so on.

Prepare busysbox

In order to avoid problems such as replacing system commands and preloading dynamic libraries, download the statically linked version of busybox to perform the survey. Or download the source code to compile the busybox source code, and note that static links are used when compiling.

Network statu

View the tcp and udp ports of network monitoring and the corresponding process information: busybox netstat-tulnp

View all network connections on the network: busybox netstat-anp

Assist in locating abnormal processes through network monitoring and network connections

Note that connections can be hidden if the attacker gets Root privileges and is implanted into the kernel or system-level Rootkit.

Process information

If an exception is found in the system, there is a high probability that there is an exception process executing.

View process information through ps

Busybox ps / ps-aux / ps-ef

Filter out some normal processes through grep-v, and then troubleshoot abnormal processes one by one

Use the top command to view processes with abnormal cpu/ memory footprint

Busybox top

Find the hidden processes in ps, and compare the process id in proc with the process id in ps to determine whether there are some processes in proc but not in ps

Ps-ef | awk'{print $2}'| sort-n | uniq > ps.pls / proc | sort-n | uniq > proc.pdiff ps.p proc.p

Execute pstree to view the process tree: pstree-p

Note that if the attacker gets Root privileges and is implanted into the kernel or system-level Rootkit, the process can be hidden more thoroughly. Some extensions have been made in reference [1] for readers' reference.

Locate malicious files

First execute busybox stat / usr/bin/ls, busybox stat / usr/bin/lsof, busybox stat / usr/bin/stat to make sure that these files have not been modified

Ls

Check the readable and writable execution directory

Ls-alt / tmp/; ls-alt / var/tmp; ls-alt / dev/shm

Sort the files of the directory under the $PATH environment variable, such as

Ls-alt / bin, ls-alt / sbin, ls-alt / usr/bin, ls-alt / usr/sbin, etc.

Recursively view all files

Ls-aR

Stat

For any available file, view each point in time through the stat command.

Lsof

In addition, you can view it jointly with the lsof command. The common options for lsof is as follows

Lsof lists all process calls

Lsof abc.txt shows the process of opening the file abc.txt

Lsof-c abc displays the files now opened by the abc process

Lsof-p 1234 lists the files opened by the process with process number 1234

Lsof-g gid shows the process that belongs to the gid

Lsof + d / usr/local/ displays the files opened by the process in the directory

Lsof + D / usr/local/ is the same as above, but will search the directory under the directory for a long time.

Lsof-d 4 shows processes using fd 4

Lsof-I: port checks which process uses this port

Lsof-I is used to display eligible processes.

Find

Use the find command to find recent new / modified files

For example, to find JSP files that have been modified within 24 hours

The last modification occurred from 24 hours to 24 hours from the current time find. /-mtime 0-name "* .jsp"

Find new files within 72 hours

Find /-ctime-2

Find files with special permissions

Find / * .jsp-perm 4777

Diff

Use the diff command to compare the important directories and compare the differences between the invading environment and the pure environment.

For example, copy all the important directories of the environment to PC-x, and compare the two directories with the following command

Diff-r {dir 1} {dir 2} analyze malicious programs

If an illegal process is found, run ls-l / proc/$PID/exe or file / proc/$PID/exe ($PID is the pid of the abnormal process), and check the process file path corresponding to pid.

Run cat / proc/$PID/cmdline to view the commands and parameters executed by the process

Check the file types of malicious programs through the file command, such as file / tmp/.sh

If it is an ELF file, you can check the strings in the ELF through strings, which may reveal some information, such as stirngs / tmp/.elf

If a malicious program is deleted, it can be exported from memory by means of memory dump.

Restore deleted files cp / proc/ [pid] / exe / tmp/malware.dump export process memory cat / proc/ [pid] / maps7ff48bb5d000-7ff48bb5e000gdb-- pid [pid] dump memory / tmp/malware.dump 0x7ff48bb5d000 0x7ff48bb5e000 from memory copy

Check the Access,Modify,Change time of the malicious program through the stat command to find out when the system was invaded.

Suspicious malicious programs or programs with memory dumps can be uploaded to virustotal for virus scanning

Other commands that may be used, such as strings, strace, lsattr, chattr-I, getfacl,setfacl, etc.

Rootkit automatic inspection and killing

Chkrootkit

How to use it:

Wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gztar zxvf chkrootkit.tar.gzcd chkrootkit-0.53make sense./chkrootkit

Rkhunter

How to use it:

When I tested wget https://nchc.dl.sourceforge.net/project/rkhunter/rkhunter/1.4.4/rkhunter-1.4.4.tar.gz, I found that the link above could not be downloaded, so I changed the link below, wget https://fossies.org/linux/privat/rkhunter-1.4.6.tar.gztar-zxvf rkhunter-1.4.6.tar.gzcd rkhunter-1.4.6./installer.sh-- installrkhunter-c attack to maintain the view history command.

Busybox cat / .bash_history

Detect dynamic library hijacking

View environment variable dynamic library hijacking

Busybox echo $LD_PRELOAD

View configuration file dynamic library hijacking

Busybox cat / etc/ld.so.preload

If you are not sure whether the dynamic library is malicious, you can upload the dynamic library to virustotal for testing.

View Linux account

Busybox cat / etc/passwd | grep-v nologin

Busybox cat / etc/shadow

Busybox stat / etc/passwd

Busybox cat / etc/sudoers

View the account records of the server's recent login: last

Boot up

Go through the series directories and files at the beginning of init in the / etc/ directory, and the series directories and files at the beginning of rc

View the files in the / etc/init.d/ directory

Query system services, especially those that start automatically

Chkconfig-list

Service-status-all

Scheduled task

Focus on the contents of the directories and files listed below

/ etc/crontab

/ etc/cron.d/*

/ etc/cron.daily/*

/ etc/cron.hourly/*

/ etc/cron.monthly/*

/ etc/cron.weekly/

/ etc/anacrontab

/ var/spool/cron/*

/ var/spool/anacron/*

List the scheduled tasks of the current user through crontab-l

Kernel driver

Check the kernel module load: lsmod

Ssh troubleshooting

Check the / root/.ssh directory to see if there is a public key, and look at the known_hosts file to see which hosts the local machine has connected to through ssh. It is quite possible that some of these hosts have also been compromised.

Weak password / default password for intrusion reason

First of all, check the services that are open to the public through netstat to confirm whether these services (such as mysql,redis,zookeeper,tomcat, etc.) are configured with authentication, and whether the authentication uses a weak password or default password.

Check the log information of these services to see if there are any intrusion records.

View the log

Logs include system logs and application logs. System logs are stored in the / var/log directory. Application logs depend on the specific configuration of the application.

System logs include

/ var/log/cron records logs related to scheduled tasks of the system.

/ var/log/cups records the log of printed information

/ var/log/dmesg records the information of kernel self-test when the system is powered on.

/ var/log/mailog records email information

/ var/log/message logs important information of the system

/ var/log/btmp records an error login log. To use the lastb command to view

/ var/log/lastlog keeps a log of the last login time of all users in the system. To use the lastlog command to view

/ var/log/wtmp permanently records the login and logout information of all users, as well as the startup, restart and shutdown events of the system. To use the last command to view

/ var/log/utmp records the information of users who are currently logged in. To use the wdepartment who command to view the

/ var/log/secure records authentication and authorization information, such as SSH login, su switching users, sudo authorization

View ssh login record

Less / var/log/secure | grep 'Accepted' malicious process association

In most cases, the parent process of a malicious process is 1, while in some cases, the parent process of a malicious process may not be 1, for example, the parent process is httpd. In this case, you can boldly guess that the attacker achieved the attack by exploiting the vulnerability of the parent process.

View the parent process pid, that is, ppid, of the process by ps-ef

Look at users who start malicious processes through ps auxef, and if they find that, for example, a mysql user started, then it can be inferred that it was intruded through the mysql service.

System reinforcement

Change the password of each service open to me

Restrict the services that are open to the public. If it is not convenient to operate, then restrict the accessible hosts through iptables.

Upgrade middleware used by system components or services

Thank you for your reading, the above is the content of "how to deal with Linux emergency response". After the study of this article, I believe you have a deeper understanding of how to deal with Linux emergency response, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.