Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use Kdump to check Linux kernel crashes. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Kdump is a way to get a crashed Linux kernel dump, but it can be difficult to find documentation explaining its use and internal structure. In this article, I'll look at the basic usage of kdump and how kdump/kexec is implemented in the kernel.

Kexec is a Linux kernel-to-kernel boot loader that helps boot from the context of * kernels to the second kernel. Kexec shuts down * * cores, bypasses the BIOS or firmware phase, and jumps to the second kernel. As a result, restarting becomes faster without the BIOS phase.

Kdump can be used with kexec applications-for example, a second kernel starts when * cores crash, the second kernel is used to copy memory dumps of * cores, and tools such as gdb and crash can be used to analyze the cause of the crash. (in this article, I will use the term "* kernel" as the currently running kernel, "second kernel" as the kernel running with kexec, and "capture kernel" to refer to the kernel running when the current kernel crashes.)

The kexec mechanism has components in both the kernel and user space. The kernel provides several system calls for the kexec restart function. A user-space tool called kexec-tools uses these calls and provides executables to load and boot the "second kernel". Some distributions also add wrappers to kexec-tools, which helps capture and save dumps of various dump target configurations. In this article, I'll use a tool called distro-kexec-tools to avoid confusion between upstream kexec tools and distribution-specific kexec-tools code. My example will use the Fedora Linux distribution.

Fedora kexec-tools tool

Use the dnf install kexec-tools command to install fedora-kexec-tools on the Fedora machine. After installing fedora-kexec-tools, you can execute the systemctl start kdump command to start the kdump service. When this service starts, it creates a root file system (initramfs) that contains resources to be mounted to the target location to save the vmcore, as well as commands to copy and dump the vmcore to the target location. The service then loads the kernel and initramfs into the appropriate location within the crashed kernel area so that they can be executed in the event of a kernel crash.

The Fedora wrapper provides two user profiles:

/ etc/kdump.conf specifies that the configuration parameters of initramfs need to be rebuilt after modification. For example, if you change the dump target from a local disk to a NFS-mounted disk, you need the NFS-related kernel modules loaded by the "capture kernel".

/ etc/sysconfig/kdump specifies that the configuration parameters of initramfs do not need to be rebuilt after modification. For example, if you only need to modify the command-line arguments passed to "capture the kernel", you do not need to rebuild initramfs.

If the kernel fails after the kdump service starts, the "capture kernel" executes, which further executes the vmcore save process in initramfs and then reboots to the stable kernel.

Kexec-tools tool

Compiling the source code of kexec-tools results in an executable file called kexec. This executable of the same name can be used to load and execute the "second kernel", or to load the "capture kernel", which can be executed in the event of a kernel crash.

The command to load the second kernel:

# kexec-l kernel.img-- initrd=initramfs-image.img-reuse-cmdline

-- the reuse-command parameter means that the same command line is used as the "* kernel". Use-- initrd to pass initramfs. -l indicates that you are loading the "second kernel", which can be executed by the kexec application itself (kexec-e). A kernel loaded with-l cannot be executed when the kernel crashes. In order to load a "capture kernel" that can be executed in the event of a kernel crash, you must pass the argument-p instead of-l.

Load the command to capture the kernel:

# kexec-p kernel.img-- initrd=initramfs-image.img-reuse-cmdline

Echo c > / pros/sysrq-trigger can be used to crash the kernel for testing. For more information about the options provided by kexec-tools, see man kexec. Before moving on to the next section, take a look at this kexec_dump demo:

Kdump: end-to-end flow

The following figure shows the flow chart. Crashkernel memory must be reserved for kernel capture during booting of the "* kernel". You can pass crashkernel=Y@X on the kernel command line, where @ X is optional. Crashkernel=256M is suitable for most x86x64 systems; however, choosing the appropriate memory for a crashed kernel depends on many factors, such as kernel size and initramfs, as well as the memory requirements of the modules contained in initramfs and the application runtime. For more methods of passing crashed kernel parameters, see the kernel-parameters documentation.

Pratyush_f1.png

You can pass the kernel and initramfs images to the kexec executable, as shown in the command in the (kexec-tools) section. "capture the kernel" can be the same as the "* kernel" or different. Usually, the same is fine. Initramfs is optional; for example, when the kernel is compiled with CONFIG_INITRAMFS_SOURCE, you don't need it. In general, save a different capture initramfs from * initramfs, because you can get better results by automatically executing a copy of vmcore in the capture initramfs. When kexec is executed, it also loads elfcorehdr data and purgatory executables. Purgatory is a boot loader that is customized for kdump. It has been given the odd name "purgatory". It should be just a joke. Elfcorehdr has information about the memory organization of the system, while purgatory can execute and verify that the second phase of the binary or data has the correct SHA before the "capture kernel" executes. Purgatory is also optional.

When the "* * kernel" crashes, it performs the necessary exit process and switches to purgatory, if any. Purgatory validates the SHA256 that loads the binaries and, if correct, passes control to capture Kernel. Capture Kernel creates a vmcore based on system memory information received from elfcorehdr. Therefore, after the capture kernel starts, you will see a dump of the "* kernel" in / proc/vmcore. Depending on the initramfs you are using, you can now analyze the dump, copy it to any disk, or copy it automatically, and then reboot to a stable kernel.

Kernel system call

The kernel provides two system calls: kexec_load () and kexec_file_load (), which can be used to load the "second kernel" when kexec-l is executed. It also provides an additional flag for the reboot () system call that can be used to boot to the "second kernel" using kexec-e.

Kexec_load (): the kexec_load () system call loads a new kernel that can be executed later via reboot (). The prototype is defined as follows:

Long kexec_load (unsigned long entry, unsigned long nr_segments, struct kexec_segment * segments, unsigned long flags)

User space needs to pass different segments for different components, such as kernel, initramfs, etc. Therefore, the kexec executable helps to prepare these segments. The structure of kexec_segment is as follows:

Struct kexec_segment {void * buf; / * user space buffer * / size_t bufsz; / * buffer length in user space * / void * mem; / * physical address of the kernel * / size_t memsz; / * physical address length * /}

When reboot () is called with LINUX_REBOOT_CMD_KEXEC, it boots into the kernel loaded by kexec_load. If the flag KEXEC_ON_CRASH is passed to kexec_load (), the loaded kernel will not be started with reboot (LINUX_REBOOT_CMD_KEXEC); instead, it will be executed in the event of a kernel crash. CONFIG_KEXEC must be defined to use kexec, and CONFIG_CRASH_DUMP must be defined for kdump.

Kexec_file_load (): as a user, you only need to pass two parameters (kernel and initramfs) to the kexec executable. Kexec then reads the data from sysfs or other kernel information sources and creates all the segments. So using kexec_file_load () simplifies user space, passing only the kernel and initramfs file descriptors. The rest is done by the kernel itself. CONFIG_KEXEC_FILE should be enabled when using this system call. Its prototype is as follows:

Long kexec_file_load (int kernel_fd, int initrd_fd, unsigned long cmdline_len, const char _ user * cmdline_ptr, unsigned long flags)

Note that kexec_file_load can also accept the command line, but not kexec_load (). The kernel accepts and executes the command line according to different system architectures. Therefore, in the case of kexec_load (), kexec-tools passes the command line through one of the segments, such as dtb or ELF boot comments, and so on.

Currently, kexec_file_load () only supports x86 and PowerPC.

What happens when the kernel crashes?

When * cores crash, the following actions are performed before control is passed to purgatory or "capture the kernel":

Prepare the CPU register (see crash_setup_regs () in kernel code)

Update vmcoreinfo comments (see crash_save_vmcoreinfo ())

Close the non-crash CPU and save the prepared registers (see machine_crash_shutdown () and crash_save_cpu ())

You may need to disable the interrupt controller here

*, which performs a kexec restart (see machine_kexec ()), which loads or refreshes the kexec segment into memory and passes control to the execution file that enters the segment. The input segment can be the purgatory or start address of the next kernel.

ELF program header

Most of the dump cores involved in kdump are in ELF format. Therefore, it is important to understand the ELF program header, especially if you want to find the problem that vmcore prepares. Each ELF file has a program header:

Read by the system loader

Describes how to load a program into memory

You can use Objdump-p elf_file to view the program header.

An example of the ELF header for vmcore is as follows:

# objdump-p vmcore vmcore: file format elf64-littleaarch74 Program Header: NOTE off 0x0000000000010000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2 / 0 filesz 0x00000000000013e8 memsz 0x00000000000013e8 flags-LOAD off 0x0000000000020000 vaddr 0xffff000008080000 paddr 0x0000004000280000 align 2 / 0 filesz 0x0000000001460000 memsz 0x0000000001460000 flags rwx LOAD off 0x0000000001480000 vaddr 0xffff800000200000 paddr 0x0000004000200000 align 2 / 0 filesz 0x000000007fc00000 memsz 0x000000007fc00000 flags rwx LOAD off 0x0000000081080000 vaddr 0xffff8000ffe00000 paddr 0x00000040ffe00000 align 2 / 0 filesz 0x00000002fa7a0000 memsz 0x00000002fa7a0000 flags rwx LOAD off 0x000000037b820000 vaddr 0xffff8003fa9e0000 paddr 0x00000043fa9e0000 align 2 / 0 filesz 0x0000000004fc0000 memsz 0x0000000004fc0000 flags rwx LOAD off 0x00000003807e0000 vaddr 0xffff8003ff9b0000 paddr 0x00000043ff9b0000 align 2 / 0 filesz 0x0000000000010000 memsz 0x0000000000010000 flags rwx LOAD off 0x00000003807f0000 vaddr 0xffff8003ff9f0000 paddr 0x00000043ff9f0000 align 2. 0 filesz 0x0000000000610000 memsz 0x0000000000610000 flags rwx

In this example, there is a note segment and the rest are load segments. The note section provides information about CPU, and the load section provides information about replicated system memory components.

Vmcore starts with elfcorehdr, which has the same structure as the ELF header. See the representation of elfcorehdr in the following figure:

Pratyush_f2.png

Kexec-tools reads / sys/devices/system/cpu/cpu%d/crash_notes and prepares the header for CPU PT_NOTE. Similarly, it reads / sys/kernel/vmcoreinfo and prepares the header of vmcoreinfo PT_NOTE, reads system memory from / proc/iomem and prepares the memory PT_LOAD header. When the capture kernel receives the elfcorehdr, it reads the data from the address mentioned in the header and prepares the vmcore.

Crash note

Crash notes is the area of each CPU used to store CPU state in the event of a system crash; it has information about the current PID and CPU registers.

Vmcoreinfo

The note segment has a variety of kernel debugging information, such as structure size, symbol value, page size, and so on. These values are parsed by the capture kernel and embedded in / proc/vmcore. Vmcoreinfo is mainly used by makedumpfile applications. In the Linux kernel, the include/linux/kexec.h macro defines a new vmcoreinfo. Some sample macros are as follows:

VMCOREINFO_PAGESIZE ()

VMCOREINFO_SYMBOL ()

VMCOREINFO_SIZE ()

VMCOREINFO_STRUCT_SIZE ()

Makedumpfile

Much of the information in vmcore, such as available pages, is useless. Makedumpfile is an application used to exclude unnecessary pages, such as:

Pages filled with zeros

Cached pages without private flags (non-dedicated cache)

Cached pages with private flags (dedicated cache)

User process data page

Available pages.

In addition, makedumpfile compresses / proc/vmcore data during replication. It can also remove sensitive symbolic information from the dump; however, in order to do so, it first needs kernel debugging information. The debug information comes from VMLINUX or vmcoreinfo, and its output can be in ELF format or kdump compressed format.

Typical usage:

# makedumpfile-l-message-level 1-d 31 / proc/vmcore makedumpfilecore

See man makedumpfile for more information.

Kdump debugging

Problems that beginners may encounter when using kdump:

Kexec-p kernel_image did not succeed

Check to see if crash memory is allocated.

Cat / sys/kernel/kexec_crash_size should not have a zero value.

Cat / proc/iomem | grep "Crash kernel" should have an assigned range.

If not, the correct crashkernel= parameter is passed on the command line.

If not, pass the argument-d in the kexec command and send the output to the kexec-tools mailing list.

After a message from "* kernel", nothing can be seen on the console (such as "bye")

Check that the kexec-l kernel_image command after kexec-e is working.

There may be a lack of supported architectures or machine-specific options.

It is possible that the SHA verification of purgatory failed. If your architecture does not support the console in purgatory, it is difficult to debug.

It may be that the "second kernel" has already crashed.

Pass your system's earlycon or earlyprintk options to the command line of the second kernel.

Use kexec-tools mailing lists to share * * kernels and capture kernel dmesg logs.

Resources

Fedora-kexec-tools

GitHub warehouse: git://pkgs.fedoraproject.org/kexec-tools

Mailing list: kexec@lists.fedoraproject.org

Description: Specs files and scripts provide user-friendly commands and services so that kexec-tools can be automated in different user scenarios.

Kexec-tools

GitHub warehouse: git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git

Mailing list: kexec@lists.infradead.org

Description: use kernel system calls and provide the user command kexec.

Linux kernel

GitHub warehouse: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Mailing list: kexec@lists.infradead.org

Description: implements kexec_load (), kexec_file_load (), reboot () system calls, and architecture-specific code, such as machine_kexec () and machine_crash_shutdown ().

Makedumpfile

GitHub warehouse: git://git.code.sf.net/p/makedumpfile/code

Mailing list: kexec@lists.infradead.org

Description: compress and filter unnecessary components from the dump file.

This is the end of the article on "how to use Kdump to check Linux kernel crashes". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.