Shulou(Shulou.com)06/01 Report--

This article mainly introduces "what is the method of Linux system analysis of deadlock". In the daily operation, I believe that many people have doubts about the method of Linux system analysis of deadlock. The editor consulted all kinds of data and sorted out simple and useful operation methods. I hope it will be helpful for you to answer the doubt of "what is the method of Linux system analysis of deadlock"! Next, please follow the editor to study!

Four necessary conditions for deadlock generation

(1) Mutual exclusion condition: a resource can only be used by one process (thread) at a time.

(2) request and retention conditions: when a process (thread) is blocked by a request for resources, it holds on to the resources it has acquired.

(3) No deprivation condition: the resources that have been obtained by this process (thread) cannot be forcibly deprived until they are used up.

(4) Loop waiting condition: a loop waiting resource relationship is formed between multiple processes (threads).

Figure 1. Schematic diagram of a deadlock with a cross lock:

Note: after executing func2 and func4, child thread 1 acquires lock An and is trying to acquire lock B, but child thread 2 acquires lock B and is trying to acquire lock A, so child thread 1 and child thread 2 will have no way to get lock An and lock B, because they are occupied by each other and will never be released, so deadlock occurs.

Using pstack and gdb tools to analyze deadlock programs A brief introduction of pstack on Linux platform

Pstack is a very useful tool under Linux (such as Red Hat Linux system, Ubuntu Linux system, etc.). Its function is to print out the stack information of this process. You can output the call relationship stack for all threads.

A brief introduction of gdb on Linux platform

GDB is a powerful program debugging tool under UNIX released by GNU open source organization. The Linux system contains the GNU debugger gdb, which is a debugger used to debug C and C++ programs. It enables program developers to observe the internal structure of the program and the use of memory while the program is running.

Some of the main features provided by gdb are as follows:

1 run the program and set the parameters and environment that can affect the operation of the program

2 the control program stops running under the specified conditions

3 when the program stops, you can check the status of the program

4 when the program crash, you can check the core file

5 you can correct the errors of the program and rerun the program

6 you can dynamically monitor the value of variables in the program

7 you can step through the code to observe the running status of the program.

The object of gdb program debugging is an executable file or process, not the source code file of the program. However, not all executables can be debugged with gdb. If you want to make the resulting executable available for debugging, you need to specify that the program contains debugging information at compile time by adding the-g parameter to the compiler when executing the Graph + (gcc) instruction. The debugging information contains the type of each variable in the program and the address mapping in the executable file, as well as the line number of the source code. Gdb uses this information to associate source code with machine code. There are many basic commands of gdb, which are not described in detail. If you need to know more, please refer to the gdb manual.

Listing 1. Test program

# include # include # include pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t mutex2 = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t mutex3 = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t mutex4 = PTHREAD_MUTEX_INITIALIZER; static int sequence1 = 0; static int sequence2 = 0; int func1 () {pthread_mutex_lock (& mutex1); + + sequence1; sleep (1); pthread_mutex_lock (& mutex2); + + sequence2 Pthread_mutex_unlock (& mutex2); pthread_mutex_unlock (& mutex1); return sequence1;} int func2 () {pthread_mutex_lock (& mutex2); + + sequence2; sleep (1); pthread_mutex_lock (& mutex1); + + sequence1; pthread_mutex_unlock (& mutex1); pthread_mutex_unlock (& mutex2); return sequence2 } void* thread1 (void* arg) {while (1) {int iRetValue = func1 (); if (iRetValue = = 100000) {pthread_exit (NULL);} void* thread2 (void* arg) {while (1) {int iRetValue = func2 () If (iRetValue = = 100000) {pthread_exit (NULL);} void* thread3 (void* arg) {while (1) {sleep (1); char szBuf [128,128]; memset (szBuf, 0, sizeof (szBuf)); strcpy (szBuf, "thread3") } void* thread4 (void* arg) {while (1) {sleep (1); char szBuf [128]; memset (szBuf, 0, sizeof (szBuf)); strcpy (szBuf, "thread3");} int main () {pthread_t tid [4] If (pthread_create (& tid [0], NULL, & thread1, NULL)! = 0) {_ exit (1);} if (pthread_create (& tid [1], NULL, & thread2, NULL)! = 0) {_ exit (1) } if (& tid [2], NULL, & thread3, NULL)! = 0) {_ exit (1);} if (pthread_create (& tid [3], NULL, & thread4, NULL)! = 0) {_ exit (1);} sleep (5); / / pthread_cancel (tid [0]) Pthread_join (tid [0], NULL); pthread_join (tid [1], NULL); pthread_join (tid [2], NULL); pthread_join (tid [3], NULL); pthread_mutex_destroy (& mutex1); pthread_mutex_destroy (& mutex2); pthread_mutex_destroy (& mutex3); pthread_mutex_destroy (& mutex4); return 0;} listing 2. Compile test program

[dyu@xilinuxbldsrv purify] $dyu@xilinuxbldsrv purify +-g lock.cpp-o lock-lpthread listing 3. Find the process number of the test program

[dyu@xilinuxbldsrv purify] $ps-ef | grep lock dyu 6721 5751 0 15:21 pts/3 00:00:00. / lock listing 4. Output of pstack (pstack-process number) executed for the first time on a deadlock process

[dyu@xilinuxbldsrv purify] $pstack 6721 Thread 5 (Thread 0x41e37940 (LWP 6722)): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a9b in func1 () () # 4 0x0000000000400ad7 in thread1 (void*) () # 5 0x0000003d1a80673d in start_thread () From / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 4 (Thread 0x42838940 (LWP 6723)): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a17 in func2 () () # 4 0x0000000000400a53 in thread2 (void*) () # 5 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 3 (Thread 0x43239940 (LWP 6724)): # 0 0x0000003d19c9a541 in nanosleep () from / lib64/libc.so.6 # 1 0x0000003d19c9a364 in sleep () from / lib64/libc.so.6 # 2 0x00000000004009bc in thread3 (void*) () # 3 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 4 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 2 (Thread 0x43c3a940 (LWP 6725)): # 0 0x0000003d19c9a541 in nanosleep () from / lib64/libc.so.6 # 1 0x0000003d19c9a364 in sleep () from / lib64/libc.so.6 # 2 0x0000000000400976 in thread4 (void*) () # 3 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 4 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 1 (Thread 0x2b984ecabd90 (LWP 6721) ): # 0 0x0000003d1a807b35 in pthread_join () from / lib64/libpthread.so.0 # 1 0x0000000000400900 in main () listing 5. Output of the second execution of pstack (pstack-process number) on the deadlock process [dyu@xilinuxbldsrv purify] $pstack 6721 Thread 5 (Thread 0x40bd6940 (LWP 6722)): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a87 in func1 ( ) () # 4 0x0000000000400ac3 in thread1 (void*) () # 5 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 4 (Thread 0x415d7940 (LWP 6723): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock ( ) from / lib64/libpthread.so.0 # 3 0x0000000000400a03 in func2 () # 4 0x0000000000400a3f in thread2 (void*) () # 5 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 3 (Thread 0x41fd8940 (LWP 6724)): # 0 0x0000003d19c7aec2 in memset () from / lib64/libc.so.6 # 1 0x00000000004009be in thread3 (void*) () # 2 0x0000003d1a80673d in start_thread ( ) from / lib64/libpthread.so.0 # 3 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 2 (Thread 0x429d9940 (LWP 6725)): # 0 0x0000003d19c7ae0d in memset () from / lib64/libc.so.6 # 1 0x0000000000400982 in thread4 (void*) () # 2 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 3 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 1 (Thread 0x2af906fd9d90 (LWP 6721)): # 0 0x0000003d1a807b35 in Pthread_join () from / lib64/libpthread.so.0 # 1 0x0000000000400900 in main () listing 5. The output of the second execution of pstack (pstack-process number) on the deadlock process

[dyu@xilinuxbldsrv purify] $pstack 6721 Thread 5 (Thread 0x40bd6940 (LWP 6722)): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a87 in func1 () () # 4 0x0000000000400ac3 in thread1 (void*) () # 5 0x0000003d1a80673d in start_thread () From / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 4 (Thread 0x415d7940 (LWP 6723)): # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a03 in func2 () () # 4 0x0000000000400a3f in thread2 (void*) () # 5 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 3 (Thread 0x41fd8940 (LWP 6724)): # 0 0x0000003d19c7aec2 in memset () from / lib64/libc.so.6 # 1 0x00000000004009be in thread3 (void*) () # 2 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 3 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 2 (Thread 0x429d9940 (LWP 6725)): # 0 0x0000003d19c7ae0d in memset () from / lib64/libc.so.6 # 1 0x0000000000400982 in thread4 (void*) () # 2 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 3 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 Thread 1 (Thread 0x2af906fd9d90 (LWP 6721)): # 0 0x0000003d1a807b35 in pthread_join () from / lib64/libpthread.so.0 # 1 0x0000000000400900 in main () displays details

Check the function call relationship stack of this process for analysis: when the process hangs, use pstack to check the function call stack of the process many times, the deadlock thread will always be in the state of waiting for lock, compare the output results of the function call stack many times to determine which two threads (or several threads) have been unchanged and have been in the state of waiting for lock (there may be two threads that have not changed all the time).

Output analysis:

According to the output comparison above, it can be found that thread 1 and thread 2 change from the sleep function of the first pstack output to the memset function of the second pstack output. However, thread 4 and thread 5 have been in an equal lock state (pthread_mutex_lock), and there is no change in two consecutive pstack information output, so we can speculate that thread 4 and thread 5 have a deadlock.

Gdb into thread`` output: listing 6. Then go to the deadlock process via gdb attach

(gdb) info thread 5 Thread 0x41e37940 (LWP 6722) 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 4 Thread 0x42838940 (LWP 6723) 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 3 Thread 0x43239940 (LWP 6724) 0x0000003d19c9a541 in nanosleep () from / lib64/libc.so.6 2 Thread 0x43c3a940 (LWP 6725) 0x0000003d19c9a541 in nanosleep () from / lib64/libc.so.6 * 1 Thread 0x2b984ecabd90 (LWP 6721) 0x0000003d1a807b35 in pthread_join () from / lib64/libpthread.so.0 listing 7. Switch to the output of thread 5

(gdb) thread 5 [Switching to thread 5 (Thread 0x41e37940 (LWP 6722))] # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 (gdb) where # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 # 1 0x0000003d1a808e1a in _ L_lock_1034 () from / lib64/libpthread.so.0 # 2 0x0000003d1a808cdc in pthread_mutex_lock () from / lib64/libpthread.so.0 # 3 0x0000000000400a9b in func1 () at lock.cpp:18 # 4 0x0000000000400ad7 in thread1 (arg=0x0) at lock.cpp:43 # 5 0x0000003d1a80673d in start_thread () from / lib64/libpthread.so.0 # 6 0x0000003d19cd40cd in clone () from / lib64/libc.so.6 listing 8. Output of thread 4 and thread 5

(gdb) f 3 # 3 0x0000000000400a9b in func1 () at lock.cpp:18 18 pthread_mutex_lock (& mutex2); (gdb) thread 4 [Switching to thread 4 (Thread 0x42838940 (LWP 6723))] # 0 0x0000003d1a80d4c4 in _ lll_lock_wait () from / lib64/libpthread.so.0 (gdb) f 3 # 3 0x0000000000400a17 in func2 () at lock.cpp:31 31 pthread_mutex_lock (& mutex1) (gdb) p mutex1 $1 = {_ _ data = {_ _ lock = 2, _ _ count = 0, _ _ owner = 6722, _ _ nusers = 1, _ _ kind = 0, _ _ spins = 0, _ _ list = {_ _ prev = 0x0, _ _ next = 0x0}}, _ _ size = "\ 002\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000\ 000",'\ 000' _ _ align = 2} (gdb) p mutex3 $2 = {_ _ data = {_ _ lock = 0, _ _ count = 0, _ _ owner = 0, _ _ nusers = 0, _ _ kind = 0, _ _ spins = 0, _ _ list = {_ _ prev = 0x0, _ _ next = 0x0}}, _ _ size ='\ 000cycles, _ _ align = 0} (gdb) p mutex2 $3 = {_ data = {_ lock = 2, _ _ count = 0 _ _ owner = 6723, _ _ nusers = 1, _ _ kind = 0, _ _ spins = 0, _ _ list = {_ _ prev = 0x0, _ _ next = 0x0}}, _ _ size = "\ 002\ 000\ 000\ 000\ 000\ 000\ 000C\ 032\ 000\ 000\ 00",'\ 000', _ align = 2} (gdb) can be found from above. Thread 4 is trying to acquire the lock mutex1, but the lock mutex1 has been obtained by the thread with LWP 6722 (owner = 6722), and thread 5 is trying to acquire the lock mutex2, but the lock mutex2 has been obtained by the thread with LWP 6723 (owner = 6723). From the output of pstack, we can see that LWP 6722 corresponds to thread 5 and LWP 6723 corresponds to thread 4. So we can conclude that thread 4 and thread 5 have a deadlock that crosses locks. Looking at the source code of the thread, it is found that thread 4 and thread 5 use both mutex1 and mutex2, and the application order is not reasonable.

At this point, the study of "what is the method of deadlock analysis by Linux system" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.