• Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.1.2
    • Component/s: None
    • None
    • Environment:
      "version" : "2.1.2-pre-",
      "gitVersion" : "d15b244ccabe1d8d0d1e93f2cb8de03d6b1d09d8",
    • ALL

      mongos process eat all memory (94%)

      # cat /proc/12220/status
      Name:   mongos
      State:  S (sleeping)
      Tgid:   12220
      Pid:    12220
      PPid:   1
      TracerPid:      0
      Uid:    113     113     113     113
      Gid:    65534   65534   65534   65534
      FDSize: 1024
      Groups: 120 65534 
      VmPeak: 79183616 kB
      VmSize: 78010392 kB
      VmLck:         0 kB
      VmHWM:  46019552 kB
      VmRSS:  46019552 kB
      VmData: 77986428 kB
      VmStk:        88 kB
      VmExe:      5980 kB
      VmLib:      3344 kB
      VmPTE:    151204 kB
      Threads:        82
      SigQ:   0/16382
      SigPnd: 0000000000000000
      ShdPnd: 0000000000000000
      SigBlk: 0000000000000000
      SigIgn: 0000000000000000
      SigCgt: 00000001800046e7
      CapInh: 0000000000000000
      CapPrm: 0000000000000000
      CapEff: 0000000000000000
      CapBnd: ffffffffffffffff
      Cpus_allowed:   ffffff
      Cpus_allowed_list:      0-23
      Mems_allowed:   00000000,00000003
      Mems_allowed_list:      0-1
      voluntary_ctxt_switches:        32887795
      nonvoluntary_ctxt_switches:     1434
      
      # ps u 12220
      USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
      mongodb  12220  2.6 92.8 78010920 46027844 ?   Sl   Jul05 147:46 /usr/bin/mongos --config /etc/mongos.conf --pidfilepath /work/mongo/mongos.pid
      

      While on docs.mongodb.org I read that it a process that don't need many memory

      Open files

      # ls -l /proc/12220/fd | fgrep -v socket
      total 0
      lrwx------ 1 mongodb nogroup 64 Jul  9 08:24 0 -> /dev/null
      l-wx------ 1 mongodb nogroup 64 Jul  9 08:24 1 -> /var/log/mongodb/mongos.log
      l-wx------ 1 mongodb nogroup 64 Jul  9 08:24 2 -> /var/log/mongodb/mongos.log
      lr-x------ 1 mongodb nogroup 64 Jul  9 08:24 3 -> /dev/urandom
      

      Sockets:

      # ls -l /proc/12220/fd | fgrep socket | wc -l
      318
      

      "mongod" instance is down because of this, that locate at this host too

      [1209528.992738] active_anon:11272433 inactive_anon:867823 isolated_anon:0
      [1209528.992739]  active_file:465 inactive_file:607 isolated_file:96
      [1209528.992740]  unevictable:0 dirty:8 writeback:0 unstable:0
      [1209528.992741]  free:35619 slab_reclaimable:11632 slab_unreclaimable:46570
      [1209528.992741]  mapped:2193 shmem:1089 pagetables:103091 bounce:0
      [1209528.992743] Node 0 DMA free:15736kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15152kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
      [1209528.992751] lowmem_reserve[]: 0 2991 24201 24201
      [1209528.992753] Node 0 DMA32 free:90612kB min:1736kB low:2168kB high:2604kB active_anon:1954904kB inactive_anon:498024kB active_file:428kB inactive_file:508kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:3063520kB mlocked:0kB dirty:4kB writeback:0kB mapped:2084kB shmem:56kB slab_reclaimable:7820kB slab_unreclaimable:93904kB kernel_stack:5368kB pagetables:21756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
      [1209528.992762] lowmem_reserve[]: 0 0 21210 21210
      [1209528.992764] Node 0 Normal free:14140kB min:12332kB low:15412kB high:18496kB active_anon:20224444kB inactive_anon:1445744kB active_file:476kB inactive_file:908kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:21719040kB mlocked:0kB dirty:0kB writeback:0kB mapped:2400kB shmem:1668kB slab_reclaimable:20684kB slab_unreclaimable:36336kB kernel_stack:3416kB pagetables:200632kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
      [1209528.992773] lowmem_reserve[]: 0 0 0 0
      [1209528.992775] Node 1 Normal free:22748kB min:14092kB low:17612kB high:21136kB active_anon:22910384kB inactive_anon:1527524kB active_file:956kB inactive_file:1012kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:24821760kB mlocked:0kB dirty:28kB writeback:0kB mapped:4288kB shmem:2632kB slab_reclaimable:18024kB slab_unreclaimable:56040kB kernel_stack:4688kB pagetables:189976kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
      [1209528.992783] lowmem_reserve[]: 0 0 0 0
      [1209528.992785] Node 0 DMA: 0*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15736kB
      [1209528.992792] Node 0 DMA32: 14434*4kB 4164*8kB 34*16kB 1*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 91688kB
      [1209528.992798] Node 0 Normal: 3459*4kB 6*8kB 0*16kB 2*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 14140kB
      [1209528.992804] Node 1 Normal: 5917*4kB 47*8kB 2*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24108kB
      [1209528.992810] 17720 total pagecache pages
      [1209528.992811] 14833 pages in swap cache
      [1209528.992813] Swap cache stats: add 8503273, delete 8488440, find 2030254/2098475
      [1209528.992814] Free swap  = 0kB
      [1209528.992815] Total swap = 31249400kB
      [1209529.149310] 12582896 pages RAM
      [1209529.149312] 193293 pages reserved
      [1209529.149314] 346058 pages shared
      [1209529.149315] 12349055 pages non-shared
      [1209529.149318] Out of memory: kill process 9308 (mongod) score 459992 or a child
      [1209529.149364] Killed process 9308 (mongod)
      

      Swap is also already full

      # free -m
                   total       used       free     shared    buffers     cached
      Mem:         48396      48074        322          0         15       2126
      -/+ buffers/cache:      45932       2464
      Swap:        30516      30512          4
      

      And as shows 'free -m', cached or buffers are very small.
      So maybe "mongos" have memory leak

        1. mongos-instance-memory1.png
          mongos-instance-memory1.png
          9 kB
        2. mongos-instance-memory-after-restart.png
          mongos-instance-memory-after-restart.png
          7 kB
        3. mongos_25335_20120710.pmap
          12 kB
        4. mongo_mem-month.png
          mongo_mem-month.png
          34 kB
        5. mongos.pmap
          110 kB
        6. mongos-23012_-x.pmap
          23 kB

            Assignee:
            benjamin.becker Ben Becker
            Reporter:
            azat Azat Khuzhin
            Votes:
            2 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: