Feb 11 05:16:47 slave-4 kernel: mongod invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
|
Feb 11 05:16:47 slave-4 kernel: mongod cpuset=/ mems_allowed=0-1
|
Feb 11 05:16:47 slave-4 kernel: Pid: 26834, comm: mongod Not tainted 2.6.32-431.el6.x86_64 #1
|
Feb 11 05:16:47 slave-4 kernel: Call Trace:
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff810d05b1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81122960>] ? dump_header+0x90/0x1b0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8129032d>] ? __bitmap_intersects+0x1d/0xa0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8122798c>] ? security_real_capable_noaudit+0x3c/0x70
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81122de2>] ? oom_kill_process+0x82/0x2a0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81122d21>] ? select_bad_process+0xe1/0x120
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81123220>] ? out_of_memory+0x220/0x3c0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8112fb3c>] ? __alloc_pages_nodemask+0x8ac/0x8d0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff811651c9>] ? alloc_page_interleave+0x39/0x90
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81167afc>] ? alloc_pages_current+0x10c/0x110
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8111fd57>] ? __page_cache_alloc+0x87/0x90
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8111f73e>] ? find_get_page+0x1e/0xa0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81121695>] ? generic_file_aio_read+0x585/0x700
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff81188dba>] ? do_sync_read+0xfa/0x140
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff810a07c8>] ? up_read+0x18/0x30
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8109b2a0>] ? autoremove_wake_function+0x0/0x40
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff812263c6>] ? security_file_permission+0x16/0x20
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff811896a5>] ? vfs_read+0xb5/0x1a0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff811899d2>] ? sys_pread64+0x82/0xa0
|
Feb 11 05:16:47 slave-4 kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
|
Feb 11 05:16:47 slave-4 kernel: Mem-Info:
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA per-cpu:
|
Feb 11 05:16:47 slave-4 kernel: CPU 0: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 1: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 2: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 3: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 4: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 5: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 6: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 7: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 8: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 9: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 10: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 11: hi: 0, btch: 1 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA32 per-cpu:
|
Feb 11 05:16:47 slave-4 kernel: CPU 0: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 1: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 2: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 3: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 4: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 5: hi: 186, btch: 31 usd: 30
|
Feb 11 05:16:47 slave-4 kernel: CPU 6: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 7: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 9: hi: 186, btch: 31 usd: 30
|
Feb 11 05:16:47 slave-4 kernel: CPU 10: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: Node 0 Normal per-cpu:
|
Feb 11 05:16:47 slave-4 kernel: CPU 0: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 1: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 2: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 3: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 4: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 5: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 6: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 7: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 9: hi: 186, btch: 31 usd: 29
|
Feb 11 05:16:47 slave-4 kernel: CPU 10: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: Node 1 Normal per-cpu:
|
Feb 11 05:16:47 slave-4 kernel: CPU 0: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 1: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 2: hi: 186, btch: 31 usd: 36
|
Feb 11 05:16:47 slave-4 kernel: CPU 3: hi: 186, btch: 31 usd: 28
|
Feb 11 05:16:47 slave-4 kernel: CPU 4: hi: 186, btch: 31 usd: 14
|
Feb 11 05:16:47 slave-4 kernel: CPU 5: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 6: hi: 186, btch: 31 usd: 30
|
Feb 11 05:16:47 slave-4 kernel: CPU 7: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 9: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 10: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Feb 11 05:16:47 slave-4 kernel: active_anon:23225295 inactive_anon:1240638 isolated_anon:576
|
Feb 11 05:16:47 slave-4 kernel: active_file:193 inactive_file:373 isolated_file:447
|
Feb 11 05:16:47 slave-4 kernel: unevictable:0 dirty:0 writeback:0 unstable:0
|
Feb 11 05:16:47 slave-4 kernel: free:71811 slab_reclaimable:5826 slab_unreclaimable:10322
|
Feb 11 05:16:47 slave-4 kernel: mapped:440 shmem:0 pagetables:50680 bounce:0
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA free:15488kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15076kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
|
Feb 11 05:16:47 slave-4 kernel: lowmem_reserve[]: 0 2991 48441 48441
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA32 free:184516kB min:2780kB low:3472kB high:4168kB active_anon:1515720kB inactive_anon:505176kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):128kB isolated(file):124kB present:3063584kB mlocked:0kB dirty:0kB writeback:4kB mapped:24kB shmem:0kB slab_reclaimable:864kB slab_unreclaimable:216kB kernel_stack:0kB pagetables:3992kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
|
Feb 11 05:16:47 slave-4 kernel: lowmem_reserve[]: 0 0 45450 45450
|
Feb 11 05:16:47 slave-4 kernel: Node 0 Normal free:42176kB min:42248kB low:52808kB high:63372kB active_anon:44469264kB inactive_anon:2223260kB active_file:600kB inactive_file:264kB unevictable:0kB isolated(anon):1408kB isolated(file):384kB present:46540800kB mlocked:0kB dirty:0kB writeback:0kB mapped:1116kB shmem:0kB slab_reclaimable:10580kB slab_unreclaimable:22896kB kernel_stack:2472kB pagetables:96412kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
|
Feb 11 05:16:47 slave-4 kernel: lowmem_reserve[]: 0 0 0 0
|
Feb 11 05:16:47 slave-4 kernel: Node 1 Normal free:45064kB min:45064kB low:56328kB high:67596kB active_anon:46916196kB inactive_anon:2234116kB active_file:172kB inactive_file:856kB unevictable:0kB isolated(anon):768kB isolated(file):896kB present:49643520kB mlocked:0kB dirty:84kB writeback:0kB mapped:620kB shmem:0kB slab_reclaimable:11860kB slab_unreclaimable:18176kB kernel_stack:912kB pagetables:102316kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:64 all_unreclaimable? no
|
Feb 11 05:16:47 slave-4 kernel: lowmem_reserve[]: 0 0 0 0
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA: 2*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15488kB
|
Feb 11 05:16:47 slave-4 kernel: Node 0 DMA32: 239*4kB 141*8kB 224*16kB 174*32kB 81*64kB 75*128kB 60*256kB 42*512kB 23*1024kB 6*2048kB 21*4096kB = 184740kB
|
Feb 11 05:16:47 slave-4 kernel: Node 0 Normal: 9548*4kB 131*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 43352kB
|
Feb 11 05:16:47 slave-4 kernel: Node 1 Normal: 9932*4kB 286*8kB 7*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 46224kB
|
Feb 11 05:16:47 slave-4 kernel: 142914 total pagecache pages
|
Feb 11 05:16:47 slave-4 kernel: 141768 pages in swap cache
|
Feb 11 05:16:47 slave-4 kernel: Swap cache stats: add 3496866, delete 3355098, find 1110552/1271814
|
Feb 11 05:16:47 slave-4 kernel: Free swap = 0kB
|
Feb 11 05:16:47 slave-4 kernel: Total swap = 4194296kB
|
Feb 11 05:16:47 slave-4 kernel: 25165808 pages RAM
|
Feb 11 05:16:47 slave-4 kernel: 401628 pages reserved
|
Feb 11 05:16:47 slave-4 kernel: 1489 pages shared
|
Feb 11 05:16:47 slave-4 kernel: 24685540 pages non-shared
|
Feb 11 05:16:47 slave-4 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
|
Feb 11 05:16:47 slave-4 kernel: [ 606] 0 606 2733 0 0 -17 -1000 udevd
|
Feb 11 05:16:47 slave-4 kernel: [ 965] 0 965 2731 0 3 -17 -1000 udevd
|
Feb 11 05:16:47 slave-4 kernel: [ 969] 0 969 2733 0 0 -17 -1000 udevd
|
Feb 11 05:16:47 slave-4 kernel: [ 1230] 0 1230 6910 31 8 -17 -1000 auditd
|
Feb 11 05:16:47 slave-4 kernel: [ 1246] 0 1246 62272 90 6 0 0 rsyslogd
|
Feb 11 05:16:47 slave-4 kernel: [ 1258] 81 1258 5351 1 6 0 0 dbus-daemon
|
Feb 11 05:16:47 slave-4 kernel: [ 1294] 1055 1294 30001 101 2 0 0 mqexec
|
Feb 11 05:16:47 slave-4 kernel: [ 1308] 0 1308 16651 9 6 -17 -1000 sshd
|
Feb 11 05:16:47 slave-4 kernel: [ 1316] 38 1316 7681 41 6 0 0 ntpd
|
Feb 11 05:16:47 slave-4 kernel: [ 1392] 0 1392 20318 19 0 0 0 master
|
Feb 11 05:16:47 slave-4 kernel: [ 1400] 0 1400 29325 22 0 0 0 crond
|
Feb 11 05:16:47 slave-4 kernel: [ 1403] 89 1403 20381 34 3 0 0 qmgr
|
Feb 11 05:16:47 slave-4 kernel: [ 1413] 1036 1413 35372 97 1 0 0 munin-node
|
Feb 11 05:16:47 slave-4 kernel: [ 1452] 0 1452 5385 0 0 0 0 atd
|
Feb 11 05:16:47 slave-4 kernel: [ 1780] 0 1780 70679 976 4 0 0 chef-client
|
Feb 11 05:16:47 slave-4 kernel: [ 1809] 0 1809 1020 1 0 0 0 agetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1810] 0 1810 1016 1 1 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1812] 0 1812 1016 1 3 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1814] 0 1814 1016 1 1 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1816] 0 1816 1016 1 8 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1818] 0 1818 1016 1 1 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1820] 0 1820 1016 1 2 0 0 mingetty
|
Feb 11 05:16:47 slave-4 kernel: [ 1822] 89 1822 20337 26 2 0 0 tlsmgr
|
Feb 11 05:16:47 slave-4 kernel: [ 3187] 9061 3187 6818 244 1 0 0 tmux
|
Feb 11 05:16:47 slave-4 kernel: [ 3188] 9061 3188 27109 7 1 0 0 bash
|
Feb 11 05:16:47 slave-4 kernel: [ 3277] 9061 3277 27109 7 1 0 0 bash
|
Feb 11 05:16:47 slave-4 kernel: [ 6609] 0 6609 166189 521 6 0 0 salt-minion
|
Feb 11 05:16:47 slave-4 kernel: [24919] 9061 24919 25370650 24167174 2 0 0 mongod
|
Feb 11 05:16:47 slave-4 kernel: [24934] 9061 24934 26516 38 6 0 0 run.simple.bash
|
Feb 11 05:16:47 slave-4 kernel: [25160] 89 25160 20338 17 5 0 0 pickup
|
Feb 11 05:16:47 slave-4 kernel: [26695] 9061 26695 7927842 155568 6 0 0 java
|
Feb 11 05:16:47 slave-4 kernel: [26696] 9061 26696 25228 24 4 0 0 tee
|
Feb 11 05:16:47 slave-4 kernel: Out of memory: Kill process 24919 (mongod) score 977 or sacrifice child
|
Feb 11 05:16:47 slave-4 kernel: Killed process 24919, UID 9061, (mongod) total-vm:101482600kB, anon-rss:96667388kB, file-rss:1372kB
|