Shulou(Shulou.com)06/03 Report--

Linux process deadlock location

A port in telnet was recently found when locating a service problem and could not be linked. In desperation, we can only investigate step by step.

Whether the port exists

Ss-l | grep LISTEN | grep 9999

If the port exists, you can observe the recv-q send-q on the port. If deadlock occurs, the two queues will only increase (of course, when the service processing is too slow, it will lead to packet accumulation).

Recv-Q Send-Q Local Address:Port Peer Address:Port 0 1024 *: 5200 *: * you can also use the following command to count the data of all kinds of socket status ss | awk 'BEGIN {arr ["] = 0} {arr [$1] + +} END {for (i in arr) print iJournal arr [I]}' LAST-ACK 1305ESTAB 341643State 1FIN-WAIT-1 7553CLOSING 3FIN-WAIT-2 908CLOSE-WAIT 60067

If your service is multiple processes, if there is only one process deadlock, it is easy to see that the cpu consumption of that process should be less than that of other processes, depending on the running time of the process. In the following process, the process of id=1903 is a suspected deadlock problem.

Root 1901 10 11:09? 00:00:00. / client-f.. / conf/client.ini-d

Root 1902 1901 15 11:09? 00:31:55. / client-f.. / conf/client.ini-d

Root 1903 1901 1 11:09? 00:02:25. / client-f.. / conf/client.ini-d

Root 1904 1901 15 11:09? 00:31:19. / client-f.. / conf/client.ini-d

Root 1905 1901 15 11:09? 00:31:17. / client-f.. / conf/client.ini-d

Locate where the deadlock after step-by-step cross-examination, suspected process deadlock, ok. The best way to locate is to attach to the process, and then bt can see where the process hang is.

$gdb attach 1903

# 0 0x00007f105892105e in _ _ lll_lock_wait_private () from / lib64/libc.so.6

# 1 0x00007f10588c6cad in _ L_lock_2164 () from / lib64/libc.so.6

# 2 0x00007f10588c6a67 in _ _ tz_convert () from / lib64/libc.so.6

# 3 0x00007f105890da5d in _ _ vsyslog_chk () from / lib64/libc.so.6

# 4 0x00007f105889948e in _ _ libc_message () from / lib64/libc.so.6

# 5 0x00007f105889ee66 in malloc_printerr () from / lib64/libc.so.6

# 6 0x00007f10588c6909 in tzset_internal () from / lib64/libc.so.6

# 7 0x00007f10588c6a89 in _ _ tz_convert () from / lib64/libc.so.6

# 8 0x00000000004c0917 in shift_fd (lvl=1, fmt=0x55e308 "[% s] [% d] [% s]: [server] recv SIGSEGV.pid:%d!\ n") at. / src/log_xx.cpp:95

# 9 write_log (lvl=1, fmt=0x55e308 "[% s] [% d] [% s]: [server] recv SIGSEGV.pid:%d!\ n") at. / src/log_xx.cpp:138

The above problem is caused by the fact that after the process throws the SEGV signal, it uses the non-thread-safe localtime in the method of processing the signal, and this method will be shackled.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.