Shulou(Shulou.com)05/31 Report--

Editor to share with you a sample analysis of ceph bug gdb debugging, I believe that most people do not know much about it, so share this article for your reference, I hope you can learn a lot after reading this article, let's go to know it!

Environment: ceph20.2.3 armv7 32-bit, ceph compilation environment is yocto

Problem description: testing ceph on arm development when the mds process is started, the mon process will hang up.

Ceph compiles with-g by default, which can be debugged directly with gdb.

Next, use gdb to debug mon processes. Ceph-mon is multi-process, and child thread debugging mode should be turned on when gdb debugging.

Follow-fork-mode detach-on-fork description

Parent on only tries the main process (GDB default)

Child on only tries child processes.

Parent off debugs two processes at the same time, gdb and the main process, and the child process block in the fork location

Child off debugs two processes at the same time, gdb and child processes, and the main process block is in the fork location

1. Start gdb

Root@node32:~# gdbGNU gdb (GDB) 7.12.1Copyright (C) 2017 Free Software Foundation, Inc.License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details.This GDB was configured as "arm-ap-linux-gnueabi" .Type "show configuration" for configuration details.For bug reporting instructions, please see:.Find the GDB manual and other documentation resources online at:.For help, type "help" .Type "apropos word" to search for commands related to "word".

2. Read the ceph-mon file

(gdb) file ceph-monReading symbols from ceph-mon...done.

3. Set running parameters

(gdb) set args-I node32 (gdb) (gdb) show argsArgument list to give program being debugged when it is started is "- I node32".

.4. Enable multi-process debugging mode, and gdb will block the main process

(gdb) show follow-fork-modeDebugger response to a program call of fork or vfork is "parent". (gdb) show detach-on-forkWhether gdb will detach the child of a fork is on. (gdb) set follow-fork-mode child (gdb) set detach-on-fork off (gdb)

5. Run the program

(gdb) runStarting program: / usr/bin/ceph-mon-I node32 [Thread debugging using libthread_db enabled] Using host libthread_db library "/ lib/libthread_db.so.1". [new Thread 0xb68d5ce0 (LWP 2036)] [Thread 0xb68d5ce0 (LWP 2036) exited] [New process 2037] [Thread debugging using libthread_db enabled] Using host libthread_db library "/ lib/libthread_db.so.1". [new Thread 0xb68d5ce0 (LWP 2038)] Reading symbols from / usr/lib/libtcmalloc.so.4. .. done.Reading symbols from / usr/lib/libbz2.so.1...done.Reading symbols from / lib/libz.so.1...done.Reading symbols from / usr/lib/libleveldb.so.1...done.Reading symbols from / usr/lib/libsnappy.so.1...done.Reading symbols from / usr/lib/libnss3.so...done.Reading symbols from / usr/lib/libnspr4.so...done.Reading symbols from / lib/libpthread.so.0... Done.Reading symbols from / lib/libdl.so.2...done.Reading symbols from / usr/lib/libboost_thread.so.1.63.0...done.Reading symbols from / usr/lib/libboost_random.so.1.63.0...done.Reading symbols from / lib/librt.so.1...done.Reading symbols from / usr/lib/libboost_iostreams.so.1.63.0...done.Reading symbols from / usr/lib/libboost_system.so. 1.63.0...done.Reading symbols from / usr/lib/libstdc++.so.6...done.Reading symbols from / lib/libm.so.6...done.Reading symbols from / lib/libgcc_s.so.1...done.Reading symbols from / lib/libc.so.6...done.Reading symbols from / lib/ld-linux-armhf.so.3...done.Reading symbols from / usr/lib/libunwind.so.8...done.Reading symbols from / Usr/lib/libnssutil3.so...done.Reading symbols from / usr/lib/libplc4.so...done.Reading symbols from / usr/lib/libplds4.so...done. [New Thread 0xb48c5ce0 (LWP 2039)] [New Thread 0xb40c5ce0 (LWP 2040)] [New Thread 0xb38c5ce0 (LWP 2041)] [New Thread 0xb30c5ce0 (LWP 2042)] [New Thread 0xb28c5ce0 (LWP 2043)] [New Thread 0xb20c5ce0 (LWP 2044)] [New Thread 0xb18c5ce0 (LWP 2045)] [New Thread 0xb10c5ce0 (LWP 2046)] [New process 2047]

6. View running threads

(gdb) info inferiors Num Description Executable 1 process 2033 / usr/bin/ceph-mon 2 process 2037 / usr/bin/ceph-mon * 3 / bin/bash.bash (gdb)

You can see that it is now in process 3, and now process 3 does not have a process number to indicate that it has been exit.

7. Currently, both process 1 and process 2 are in a blocking state. Switch to process 1, continue

(gdb) inferior 1 [Switching to inferior 1 [process 2033] (/ usr/bin/ceph-mon)] [Switching to thread 1.1 (Thread 0xb6ff1010 (LWP 2033))] # 0 0xb6a15648 in _ _ libc_fork () at / usr/src/debug/glibc/2.25-r0/git/sysdeps/nptl/fork.c:139warning: Source file is more recent than executable.139 pid = ARCH_FORK () (gdb) where#0 0xb6a15648 in _ libc_fork () at / usr/src/debug/glibc/2.25-r0/git/sysdeps/nptl/fork.c:139#1 0x7f6eba7c in Preforker::prefork (this=0xbeffeb70, err=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/common/Preforker.h:52#2 0x7f692058 in main (argc= Argv=0x0) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/ceph_mon.cc:500 (gdb) cContinuing. [Inferior 1 (process 2033) exited normally] (gdb) info inferiors Num Description Executable * 1 / usr/bin/ceph-mon 2 process 2037 / usr/bin/ceph-mon (gdb)

You can see that process 1 has also exited. Switch to process 2

8. Switch to process 2, and continue, process 2 is blocked, waiting for the client to send a message

9. Start the mds process on another development board

10. Mon receives the message and segments the error

[New Thread 0xb08c5ce0 (LWP 2087)] [New Thread 0xb05c5ce0 (LWP 2088)] Thread 2.8 "ms_dispatch" received signal SIGSEGV Segmentation fault. [Switching to Thread 0xb20c5ce0 (LWP 2044)] 0xb6befe1c in std::local_Rb_tree_decrement (_ _ x=0x7fc14b24) at.. / work-shared/gcc-5.4.0-r0/gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc:9898.. /. . /.. / work-shared/gcc-5.4.0-r0/gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc: No such file or directory. (gdb) Continuing.Thread 2.8 "ms_dispatch" received signal SIGSEGV Segmentation fault.raise (sig=sig@entry=11) at / usr/src/debug/glibc/2.25-r0/git/sysdeps/unix/sysv/linux/raise.c:5151 / usr/src/debug/glibc/2.25-r0/git/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt # 0 raise (sig=sig@entry=11) at / usr/src/debug/glibc/2.25-r0/git/sysdeps/unix/sysv/linux/raise.c: 51-1 0x7f970930 in reraise_fatal (signum=11) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/global/signal_handler.cc:71#2 handle_fatal_signal (signum=11) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/global/signal_handler.cc:133#3 # 4 0xb6befe1c in std::local_Rb_tree_decrement (_ _ x=0x7fc14b24) at. /.. /.. /.. / work-shared/gcc-5.4.0-r0/gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc:98#5 0x7f7e585c in std::_Rb_tree_iterator::operator-- (this=) at / usr/include/c++/5.4.0/bits/stl_tree.h:220#6 std::_Rb_tree::_M_get_insert_ Hint_unique_pos (_ _ Kwon... _ _ position=..., this=0x855265dc) at / usr/include/c++/5.4.0/bits/stl_tree.h:1924#7 std::_Rb_tree::_M_emplace_hint_unique (std::_Rb_tree_const_iterator, std::piecewise_construct_t const&, std::tuple&&, std::tuple&&) (this=this@entry=0x855265dc, _ _ pos=...) At / usr/include/c++/5.4.0/bits/stl_tree.h:2174#8 0x7f9b538c in std::map::operator [] (_ _ knight.., this=0x855265dc) at / usr/include/c++/5.4.0/bits/stl_map.h:483#9 FSMap::insert (this=this@entry=0x85526518, new_info=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/mds/FSMap.cc:794#10 0x7f7d4c94 in MDSMonitor::prepare_beacon (this=this@entry=0x85526340, op=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/MDSMonitor.cc:549#11 0x7f7da428 in MDSMonitor::prepare_update (this=this@entry=0x85526340, op=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/MDSMonitor.cc:469#12 0x7f75bd20 in PaxosService::dispatch (this=this@entry=0x85526340, op=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/PaxosService.cc:96#13 0x7f72021c in Monitor::dispatch_op (this=this@entry=0x855bc000, op=...) At / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/Monitor.cc:3605#14 0x7f721078 in Monitor::_ms_dispatch (this=this@entry=0x855bc000, m=m@entry=0x855ff980) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/Monitor.cc:3532#15 0x7f743414 in Monitor::ms_dispatch (this=0x855bc000 M=0x855ff980) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/mon/Monitor.h:905#16 0x7fb769b4 in Messenger::ms_deliver_dispatch (m=0x855ff980 This=0x855b0b00) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/msg/Messenger.h:584#17 DispatchQueue::entry (this=0x855b0c80) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/msg/simple/DispatchQueue.cc:185#18 0x7fa71d24 in DispatchQueue::DispatchThread::entry (this=) at / usr/src/debug/ceph-src/10.2.3-r0/git/src/msg/simple/DispatchQueue.h : 103-19 0xb6d55f28 in start_thread (arg=0xb20c5ce0) at / usr/src/debug/glibc/2.25-r0/git/nptl/pthread_create.c:458#20 0xb6a49968 in? () at.. / sysdeps/unix/sysv/linux/arm/clone.S:76 from / lib/libc.so.6Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)

The cause of the error can be seen that line 98 of the work-shared/gcc-5.4.0-r0/gcc-5.4.0/libstdc++-v3/src/c++98/tree.cc file caused a segment error.

After debugging, it is found that because the compiler of ceph and gcc version 5.4.0 does not match, the problem of compiler 6.3.0 is solved. When compiler 6.3.0 is compiled, ceph will no longer have pg scrub and mds crash problems when pg scrub is crash and then yocto toolchain is changed to 4.9.2.

The above is all the contents of the article "sample Analysis of ceph bug gdb debugging". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.