Shulou(Shulou.com)06/01 Report--

Environment description

OS operating system: SUSE Linux 11

Database version: MYSQL 5.6

Fault problem description

The customer reported that the MYSQL database was restarted due to insufficient memory.

Fault analysis 1. Confirm the problem according to the MYSQL alarm log:

2016-10-24 12:28:49 29226 [Note] Plugin 'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB: Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB: Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB: Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB: Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB: Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB: Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB: Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB: Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB: Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB: Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB: Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB: from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0 401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128rollback segment (s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB: 5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering after a crash using / dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash recovery finished.

2016-10-24 12:29:06 29226 [Note] Server hostname (bind-address):'*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is available.

2016-10-24 12:29:06 29226 [Note] -:: 'resolves to'::'

2016-10-24 12:29:06 29226 [Note] Server socket created on IP:':'

2016-10-24 12:29:06 29226 [Warning] 'proxies_priv' entry' @ root@tsptonldb01' ignored in-skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note] / usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19 Mutual upsql socket:' / etc/mysock/it_mgmad.sock' port: 60002 UpSQL Server

2016-10-24 12:30:21 29226 [ERROR] / usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

161024 12:31:35 mysqld_safe Number of processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659 [Warning] Buffered warning: Could not increase number of max_open_files to more than 10240 (request: 25000)

From the above information, we can see that MSYQL failed to start due to an out-of-memory error.

two。 Operating system log prompt error

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482] Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996485] CPU 0: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996486] CPU 1: hi: 0, btch: 1 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996488] CPU 0: hi: 186, btch: 31 usd: 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996489] CPU 1: hi: 186, btch: 31 usd: 5

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996491] CPU 0: hi: 186, btch: 31 usd: 30

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996492] CPU 1: hi: 186, btch: 31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] active_file:353 inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496] unevictable:0 dirty:6 writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996497] free:21800 slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498] mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated (anon): 0kB isolated (file): 0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? Yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996505] lowmem_reserve []: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB inactive_file:88kB unevictable:0kB isolated (anon): 0kB isolated (file): 0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611 all_unreclaimable? Yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996513] lowmem_reserve []: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB inactive_file:6776kB unevictable:0kB isolated (anon): 0kB isolated (file): 0kB present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? Yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996520] lowmem_reserve []: 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB 43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB 5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996556] Swap cache stats: add 5506519, delete 5470793, find 22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Free swap = 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960] [pid] uid tgid total_vm rss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006974] [571] 0571 4223 1 1-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006981] [1689] 0 1689 998 10 00 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006983] [1704] 100 1704 5444 117 00 0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006986] [1722] 0 1722 4824 62 00 0 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006988] [1725] 0 1725 1065 00 00 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006991] [10038] 0 10038 31835 168 100 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006994] [10096] 0 10096 15193 100 0 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006996] [10185] 101 10185 8608 146 00 0 hald

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006999] [10188] 0 10188 27278 00 00 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007001] [10189] 0 10189 4505 2 00 0 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007004] [10284] 0 10284 8690 00 00 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007] [10308] 0 10308 8690 17 00 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007009] [10312] 101 10312 9129 01 00 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007012] [10313] 0 10313 8690 16 00 0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007014] [12046] 0 12046 7409 8 0-17-1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007017] [12048] 0 12048 3606 00 00 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007020] [12081] 0 12081 4723 16 00 0 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007022] [12083] 0 12083 2088 17 00 0 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007025] [12140] 0 12140 4216 1 1-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007027] [12141] 0 12141 4216 10-17-1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007029] [12224] 0 12224 6759 1 10 0 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031] [12701] 0 12701 17296 28 0-17-1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007034] [12715] 0 12715 17933 01 00 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007036] [12730] 0 12730 59557 71 00 0 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007038] [12798] 0 12798 9552 28 00 0 master

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007041] [12825] 51 12825 10052 31 100 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007043] [12864] 0 12864 16711 47 10 0 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007046] [12878] 0 12878 27377 121 00 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007048] [12885] 0 12885 3697 15 00 0 cron

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007051] [12924] 0 12924 1121 21 00 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007055] [12925] 0 12925 1121 21 00 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007058] [12926] 0 12926 1121 2 00 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007061] [12927] 0 12927 1121 21 00 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007064] [12928] 0 12928 1121 21 00 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007067] [12929] 0 12929 1121 2 00 0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007071] [13064] 107 13064 5548 01 00 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007074] [13065] 107 13065 3362 01 00 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007077] [13066] 107 13066 39551 82 00 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007080] [13079] 107 13079 9863 67 00 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007083] [13083] 107 13083 31861 01 00 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007086] [13084] 107 13084 70831 2912 00 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007090] [13086] 107 13086 20207 01 00 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007092] [13094] 107 13094 57536 135 00 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007095] [13096] 107 13096 10854 00 00 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007098] [13097] 107 13097 54246 00 00 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007101] [21349] 1018 21349 2857 21 00 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007105] [12777] 1016 12777 2857 2 00 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007109] [18450] 0 18450 13384 49 10 0 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007112] [18452] 0 18452 66809 13462 10 0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007115] [18489] 107 18489 5548 01 00 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007118] [18518] 0 18518 24613 01 00 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007121] [18540] 0 18540 39584 89 10 0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007124] [18613] 0 18613 4178 19 00 0 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126] [18619] 0 18619 5548 01 00 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007129] [18620] 0 18620 3429 00 00 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007132] [18623] 0 18623 9887 123 10 0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007135] [18627] 0 18627 91696 332 00 0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007138] [18631] 0 18631 23814 1 10 0 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007141] [18635] 0 18635 10908 01 00 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007144] [18639] 0 18639 16822 19 00 0 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007147] [18645] 0 18645 51782 400 10 0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007149] [18650] 0 18650 93659 621 10 0 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007152] [18651] 0 18651 135905 13563 10 0 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007155] [18653] 0 18653 36624 00 00 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007160] [18659] 0 18659 82105 455 10 0 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007171] [18662] 0 18662 12083 01 00 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007174] [18666] 0 18666 13609 10 00 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007178] [18668] 0 18668 15774 1 10 0 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007181] [18677] 0 18677 75456 1 10 0 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007184] [18681] 0 18681 56947 10 00 python

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007187] [18686] 0 18686 39298 113 00 0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007189] [18688] 0 18688 57564 125 00 0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007192] [18692] 0 18692 38020 114 10 0 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007195] [18693] 0 18693 48186 01 00 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007198] [18718] 0 18718 10962 00 00 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007201] [21417] 0 21417 59168 142 10 0 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007204] [21420] 0 21420 57129 199 00 0 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007208] [21463] 0 21463 35883 10 00 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007211] [21464] 0 21464 13445 4 00 0 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007214] [21471] 0 21471 3221 10 00 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007217] [21501] 0 21501 118060 114 00 0 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007221] [21621] 0 21621 55974 123 00 0 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007224] [21622] 0 21622 1616 00 00 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007227] [21623] 0 21623 3505 10 00 bash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007230] [21649] 0 21649 9244 10 00 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007233] [31142] 1016 31142 1100222 219744 100 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007236] [31181] 1018 31181 1039509 163992 10 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007239] [699] 1017 699 2856 98 00 0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007242] [1625] 1017 1625 1076677 414298 10 0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007246] [1861] 1019 1861 641372 48223 10 0 java

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007284] [14971] 51 14971 10014 128 00 0 pickup

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283 or sacrifice child

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB, anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G X 3.0.13-0.27-default # 1

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351803] [] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351814] [] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351825] [] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834] [] oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351841] [] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351848] [] _ _ alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351854] [] _ _ alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351863] [] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351871] [] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351878] [] _ _ do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351884] [] handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351893] [] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351900] [] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353042]

From October 24th to November, it was found that there were several cases of running out of memory and the operating system KILL dropped MYSQLD processes.

There are three MYSQL instances on this server, and the INNODB buffer pool of each MYSQL instance is set to 2GB, but the physical memory of the operating system

There is only 4GB, and there is a synchronized JAVA process on the server. Combined with log analysis, MYSQL memory allocation is unreasonable, so it is recommended to extend server memory to 10GB.

Recommended MYSQL INNODB pool calculation method: INNODB buffer pool memory = operating system physical memory-(memory reserved for operating system 1-2GB)

-(memory of JAVA process)-(maximum number of MYSQL connections * 2MB)

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.