mongos all memory

# cat /proc/12220/status

Name:   mongos

State:  S (sleeping)

Tgid:   12220

Pid:    12220

PPid:   1

TracerPid:      0

Uid:    113     113     113     113

Gid:    65534   65534   65534   65534

FDSize: 1024

Groups: 120 65534 

VmPeak: 79183616 kB

VmSize: 78010392 kB

VmLck:         0 kB

VmHWM:  46019552 kB

VmRSS:  46019552 kB

VmData: 77986428 kB

VmStk:        88 kB

VmExe:      5980 kB

VmLib:      3344 kB

VmPTE:    151204 kB

Threads:        82

SigQ:   0/16382

SigPnd: 0000000000000000

ShdPnd: 0000000000000000

SigBlk: 0000000000000000

SigIgn: 0000000000000000

SigCgt: 00000001800046e7

CapInh: 0000000000000000

CapPrm: 0000000000000000

CapEff: 0000000000000000

CapBnd: ffffffffffffffff

Cpus_allowed:   ffffff

Cpus_allowed_list:      0-23

Mems_allowed:   00000000,00000003

Mems_allowed_list:      0-1

voluntary_ctxt_switches:        32887795

nonvoluntary_ctxt_switches:     1434

# ps u 12220

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

mongodb  12220  2.6 92.8 78010920 46027844 ?   Sl   Jul05 147:46 /usr/bin/mongos --config /etc/mongos.conf --pidfilepath /work/mongo/mongos.pid

# ls -l /proc/12220/fd | fgrep -v socket

total 0

lrwx------ 1 mongodb nogroup 64 Jul  9 08:24 0 -> /dev/null

l-wx------ 1 mongodb nogroup 64 Jul  9 08:24 1 -> /var/log/mongodb/mongos.log

l-wx------ 1 mongodb nogroup 64 Jul  9 08:24 2 -> /var/log/mongodb/mongos.log

lr-x------ 1 mongodb nogroup 64 Jul  9 08:24 3 -> /dev/urandom

# ls -l /proc/12220/fd | fgrep socket | wc -l

318

[1209528.992738] active_anon:11272433 inactive_anon:867823 isolated_anon:0

[1209528.992739]  active_file:465 inactive_file:607 isolated_file:96

[1209528.992740]  unevictable:0 dirty:8 writeback:0 unstable:0

[1209528.992741]  free:35619 slab_reclaimable:11632 slab_unreclaimable:46570

[1209528.992741]  mapped:2193 shmem:1089 pagetables:103091 bounce:0

[1209528.992743] Node 0 DMA free:15736kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15152kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

[1209528.992751] lowmem_reserve[]: 0 2991 24201 24201

[1209528.992753] Node 0 DMA32 free:90612kB min:1736kB low:2168kB high:2604kB active_anon:1954904kB inactive_anon:498024kB active_file:428kB inactive_file:508kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3063520kB mlocked:0kB dirty:4kB writeback:0kB mapped:2084kB shmem:56kB slab_reclaimable:7820kB slab_unreclaimable:93904kB kernel_stack:5368kB pagetables:21756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[1209528.992762] lowmem_reserve[]: 0 0 21210 21210

[1209528.992764] Node 0 Normal free:14140kB min:12332kB low:15412kB high:18496kB active_anon:20224444kB inactive_anon:1445744kB active_file:476kB inactive_file:908kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21719040kB mlocked:0kB dirty:0kB writeback:0kB mapped:2400kB shmem:1668kB slab_reclaimable:20684kB slab_unreclaimable:36336kB kernel_stack:3416kB pagetables:200632kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no

[1209528.992773] lowmem_reserve[]: 0 0 0 0

[1209528.992775] Node 1 Normal free:22748kB min:14092kB low:17612kB high:21136kB active_anon:22910384kB inactive_anon:1527524kB active_file:956kB inactive_file:1012kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:24821760kB mlocked:0kB dirty:28kB writeback:0kB mapped:4288kB shmem:2632kB slab_reclaimable:18024kB slab_unreclaimable:56040kB kernel_stack:4688kB pagetables:189976kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no

[1209528.992783] lowmem_reserve[]: 0 0 0 0

[1209528.992785] Node 0 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15736kB

[1209528.992792] Node 0 DMA32: 14434*4kB 4164*8kB 34*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 91688kB

[1209528.992798] Node 0 Normal: 3459*4kB 6*8kB 0*16kB 2*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14140kB

[1209528.992804] Node 1 Normal: 5917*4kB 47*8kB 2*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24108kB

[1209528.992810] 17720 total pagecache pages

[1209528.992811] 14833 pages in swap cache

[1209528.992813] Swap cache stats: add 8503273, delete 8488440, find 2030254/2098475

[1209528.992814] Free swap  = 0kB

[1209528.992815] Total swap = 31249400kB

[1209529.149310] 12582896 pages RAM

[1209529.149312] 193293 pages reserved

[1209529.149314] 346058 pages shared

[1209529.149315] 12349055 pages non-shared

[1209529.149318] Out of memory: kill process 9308 (mongod) score 459992 or a child

[1209529.149364] Killed process 9308 (mongod)

# free -m

             total       used       free     shared    buffers     cached

Mem:         48396      48074        322          0         15       2126

-/+ buffers/cache:      45932       2464

Swap:        30516      30512          4