Shulou(Shulou.com)06/01 Report--

A problem description

After upgrading a zabbix proxy from 2.4.5 to 3.0.2, it crashed when it was started.

The error message is as follows:

2367 PID:42385,exitcode/signal:11 20160508 One child process died 153246.830. Exiting... 42367:20160508:153248.904 Zabbix Proxy stopped. Zabbix 3.0.2 (revision 59540).

Adjust the log level to a higher level, set DebugLevel=4, and check the error report.

42629 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42628 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.004 signal:15 (SIGTERM). Exiting... 42634 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.004 signal:15 (SIGTERM). Exiting... 42651 signal:15 20160508 Got signal 153529.004 sender_pid:42623,sender_uid:498,reason:0. Exiting... 42659 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.004 signal:15 (SIGTERM). Exiting... 42661 signal:15 20160508 signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0. Exiting... 42655 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42663 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42664 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42666 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42665 SIGTERM sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.006. Exiting... 42667 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.006 signal:15 (SIGTERM). Exiting... 42630 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.006 signal:15 (SIGTERM). Exiting... 42668 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.006. Exiting... 42671 signal:15 20160508 Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42670 SIGTERM sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.007. Exiting... 42647 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42674 signal:15 20160508 Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting... 42680 sender_pid:42623,sender_uid:498,reason:0 20160508 Got signal 153529.007. Exiting... 42679 SIGTERM Got signal [signal:15 (SIGTERM), sender_pid:42623,sender_uid:498,reason:0]. Exiting...

9102 End of free_ipmi_handler 20160508 End of zbx_strpool_destroy 170455.283 End of free_ipmi_handler () 9102 End of free_ipmi_handler 20160508 In free_selfmon_collector 20160508 20160508 170455.284 In free_selfmon_collector () collector:0x7fa793e23000 910220160508 2014 170455.284 End of free_selfmon_collector () 91022014 20160508 In free_selfmon_collector 170455.284 End of free_selfmon_collector () 9102 2015 20160508 Switzerland 170455.284 In unload_modules () 91022015 20160508 In free_selfmon_collector (): 170455.284 End of unload_modules () 9102 Zabbix Proxy stopped 20160508 Zabbix Proxy stopped. Zabbix 3.0.2 (revision 59540). Zabbix_proxy [9152]: [file:'selfmon.c',line:375] lock failed: [22] Invalid argument

There are 4 proxy in total, and the operating system version of each proxy is CentOS6. There is a problem with the startup of this single zabbix proxy. As soon as it is started, a process dies.

Second problem solving

Check out that the kernel version of this proxy server is 3.9.8-1.el6.elrepo.x86_64, while the kernel version of the other proxy is 2.6.32-358.el6.x86_64 didn't think about the kernel version at first. I thought there was something wrong with the version of proxy or the setting of system parameters.

1. Troubleshoot semaphores under Linux

Check Zabbix's official BUG tracking platform Invalid argument. The semaphore may have reached the limit.

You can try to increase the value of the semaphore

Echo 256 40000 32 32000 > / proc/sys/kernel/sem

These four parameters are

SEMMSL the maximum value of each semaphore set that contains semaphores.

Maximum number of semaphores at the system level of SEMMNS

The maximum number of operations that can be specified by a SEMOPM semop (2) call

Minimum limit on the number of signals at the SEMMNI system level

Change through echo will invalidate after restart, you can change / etc/sysctl.conf

Kernel.sem = 250 32000 10000

Then execute sysctl-p

Proxy startup will still crash after changing parameters

two。 Troubleshoot the cause of the kernel version

Updating to latest kernel (3.10.0-327.10.1.el7.x86_64) + reboot solved it for me

The error reported originally may be something else than what affected me.

Check the post suspected to be caused by the kernel version, this proxy has been upgraded before, change the linux startup item setting the default startup kernel to 2.6.32. The problem is resolved after the restart and proxy no longer crashes. But previously, proxy 2.4.5 worked fine on the 3.9.8 kernel.

There are a lot of people who encounter this BUG, and there are several posts about this problem.

Three supplementary knowledge

This BUG involves the knowledge of semaphores under Linux. Semaphores are used to solve the problem of resource sharing in multi-process programming. The famous semaphore solution is the philosopher dining problem.

Use the ipcs command under Linux to view shared memory, semaphores, and message queues

Ipcs defaults to the-a parameter, which displays all the information

# ipcs- Shared Memory Segments-key shmid owner perms bytes nattch status 0x6c6c6536 0 root 4096 0 0x68031618 1835009 zabbix 8388583 0 0x6c030ad7 11632648 zabbixagen 600 365056 13 0x68031672 2392073 zabbix 600 8388583 0 0x78031672 2424842 zabbix 600 16777216 0 0x74031672 2457611 zabbix 600 4194304 0 0x67031672 2490380 zabbix 600 1336934400 0 0x73031672 2523149 zabbix 600 235929600 0 -Semaphore Arrays-key semid owner perms nsems 0x00000000 0 root 600 1 0x00000000 32769 root 6 00 1 0x00000000 4325378 apache 6 00 1 0x7a031618 622 595 zabbix 600 12 0x00000000 4358148 apache 1 0x7a030ad7 4292613 zabbixagen 851974 0x7a031672 851974 zabbix 12 0x7a031638 1015815 zabbix 600 12 0x7a031620 1441800 zabbix 600 12-Message Queues-key msqid owner perms used-bytes messages

Ipcs-m displays shared memory fragment information separately

Ipcs-s displays semaphores separately

Ipcs-Q displays message queues separately

# ipcs-u-Shared Memory Status-segments allocated 71pages allocated 1795473pages resident 68130pages swapped 57636Swap performance: 0 attempts 0 successes- Semaphore Status-used arrays = 16allocated semaphores = 152-Messages: Status-allocated queues = 0used headers = 0used space = 0 bytes

Reference:

Https://support.zabbix.com/browse/ZBX-10657

Http://www.cnblogs.com/forilen/p/4316358.html

Https://support.zabbix.com/browse/ZBX-3974

Http://blog.zabbix.com/mysterious-zabbix-problems-how-we-debug-them/1023/

Https://en.wikipedia.org/wiki/Semaphore_(programming)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.