crash because of "mmap failed with out of memory "

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 2.2.1
    • Component/s: Stability
    • None
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      I use mongo replicaset but suddenly it crashed. when attempt to start up with command "./mongod --dbpath=../data --replSet replset_BRG --port 28010 --logpath=../log/log --logappend --fork --directoryperdb", error occurs as follow:

      ***** SERVER RESTARTED *****
      
      
      Fri May  9 16:29:45 [initandlisten] MongoDB starting : pid=25550 port=28010 dbpath=/home/work/mongo-server/bin/../data 64-bit host=
      
      Fri May  9 16:29:45 [initandlisten] db version v2.2.1, pdfile version 4.5
      Fri May  9 16:29:45 [initandlisten] git version: d6764bf8dfe0685521b8bc7b98fd1fab8cfeb5ae
      Fri May  9 16:29:45 [initandlisten] build info: Linux domU-12-31-39-16-30-A2 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64
      BOOST_LIB_VERSION=1_49
      Fri May  9 16:29:45 [initandlisten] options: { dbpath: "../data", directoryperdb: true, fork: true, logappend: true, logpath: "../log/log"
      , port: 28010, replSet: "replset_BRG" }
      Fri May  9 16:29:45 [initandlisten] journal dir=/home/work/mongo-server/bin/../data/journal
      Fri May  9 16:29:45 [initandlisten] recover begin
      Fri May  9 16:29:45 [initandlisten] recover lsn: 0
      Fri May  9 16:29:45 [initandlisten] recover /home/work/mongo-server/bin/../data/journal/j._0
      Fri May  9 16:29:45 [initandlisten] recover cleaning up
      Fri May  9 16:29:45 [initandlisten] removeJournalFiles
      Fri May  9 16:29:45 [initandlisten] recover done
      Fri May  9 16:29:45 [initandlisten] waiting for connections on port 28010
      Fri May  9 16:29:45 [websvr] admin web console waiting for connections on port 29010
      Fri May  9 16:29:45 [initandlisten] connection accepted from 10.46.23.36:23864 #1 (1 connection now open)
      Fri May  9 16:29:45 [rsStart] replSet I am *:28010
      Fri May  9 16:29:45 [rsStart] replSet STARTUP2
      Fri May  9 16:29:45 [rsHealthPoll] replSet member*:28010 is up
      Fri May  9 16:29:45 [rsHealthPoll] replSet member *:28010 is now in state PRIMARY
      Fri May  9 16:29:45 [rsHealthPoll] replSet member *:28010 is up
      Fri May  9 16:29:45 [rsHealthPoll] replSet member *:28010 is now in state ARBITER
      Fri May  9 16:29:46 [rsSync] replSet still syncing, not yet to minValid optime 536b0dc8:1
      Fri May  9 16:29:47 [initandlisten] connection accepted from 10.42.152.12:56039 #2 (2 connections now open)
      Fri May  9 16:29:47 [conn2] end connection *:56039 (1 connection now open)
      Fri May  9 16:29:47 [initandlisten] connection accepted from 10.42.152.12:56040 #3 (2 connections now open)
      Fri May  9 16:29:47 [initandlisten] connection accepted from 10.23.240.150:59369 #4 (3 connections now open)
      Fri May  9 16:29:47 [initandlisten] connection accepted from 10.23.240.150:59374 #5 (3 connections now open)
      Fri May  9 16:29:51 [rsBackgroundSync] replSet syncing to: 10.42.152.12:28010
      Fri May  9 16:29:51 [rsSync] replSet still syncing, not yet to minValid optime 536b0dc8:1
      Fri May  9 16:29:53 [conn5] end connection *:59374 (2 connections now open)
      Fri May  9 16:29:53 [initandlisten] connection accepted from 10.23.240.150:21190 #6 (3 connections now open)
      Fri May  9 16:29:56 [rsSyncNotifier] replset setting oplog notifier to 9.42.152.12:28010
      Fri May  9 16:29:56 [repl prefetch worker] ERROR:   mmap() failed for /home/work/mongo-server/bin/../data/bridge/bridge.59 len:2146435072
      errno:12 Cannot allocate memory
      Fri May  9 16:29:56 [repl prefetch worker] ERROR: mmap failed with out of memory. (64 bit build)
      Fri May  9 16:29:56 [repl writer worker 1] ERROR:   mmap() failed for /home/work/mongo-server/bin/../data/bridge/bridge.59 len:2146435072
      errno:12 Cannot allocate memory
      Fri May  9 16:29:56 [repl writer worker 1] ERROR: mmap failed with out of memory. (64 bit build)
      Fri May  9 16:29:56 [repl writer worker 1] ERROR: writer worker caught exception: can't map file memory on: { ts: Timestamp 1399524808000|
      1, h: 1490830305013686414, v: 2, op: "i", ns: "bridge.track", o: { _id: ObjectId('536b0dc8e662b15490342d01'), bid: 515059940390023161, sit
      e: 2758320, type: 0, tm: 1399523870, date: 20140508, bdid: "BD1BCA60A411EBF3126DAAFFA7450E0B:FG=1", wt: 6, ref: "http://.com
      /brgview/?p=010b5fb9f4c42c2ea54a04de2716c98e52c8260578b2ff7d6ce067637d4b7aeef66afd58f0b9a1d5b91f2243d7e8ed138f64097a5166140bc...", url: "h
      ttp://www.55599120.com/", title: "[1m~S~H灏~T婊ㄦ~@х~W~E[1m~L婚~Y[1m~S~H灏~T婊ㄦ不[1m~V~W[1m~@х~W~E[1m~\~@濂界~Z~D[1m~L婚~Y榛~Q榫~Y姹~_[1m~\~A[1m~E[33;1H~I娑~H[1m~X叉~@婚~X~_[1m~L婚~Y~@~P[1m~E[1m~K[1m~T茬骇[1m~L婚~Y~@~Q" } }
      Fri May  9 16:29:56 [repl writer worker 1]   Fatal Assertion 16360
      0x9a28c1 0x967ea3 0x845e7e 0x97614d 0x9e82f9 0xacba3d 0xb91039
      0x9a28c1 0x967ea3 0x845e7e 0x97614d 0x9e82f9 0xacba3d 0xb91039
       [0x9a28c1]
       [0x967ea3]
       [0x845e7e]
       [0x97614d]
       [0x9e82f9]
       [0xacba3d]
       [0xb91039]
      Fri May  9 16:29:56 [repl writer worker 1]
      
      ***aborting after fassert() failure
      
      
      Fri May  9 16:29:56 Got signal: 6 (Aborted).
      Fri May  9 16:29:56 Backtrace:
      0x9a28c1 0x400c69 0xacfc00 0xacfbcd 0xb624a0 0x967ede 0x845e7e 0x97614d 0x9e82f9 0xacba3d 0xb91039
       [0x9a28c1]
       [0x400c69]
       [0xacfc00]
       [0xacfbcd]
       [0xb624a0]
       [0x967ede]
       [0x845e7e]
       [0x97614d]
       [0x9e82f9]
       [0xacba3d]
       [0xb91039]
      

      memory and disk space is enough:

       free -m
                   total       used       free     shared    buffers     cached
      Mem:         64350      64322         27          0         98      56113
      -/+ buffers/cache:       8110      56240
      Swap:          996        974         22
      
      df -hl
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/sda2             8.7G  4.3G  4.5G  49% /
      /dev/sda3             1.9T  975G  938G  51% /home
      
      

            Assignee:
            Kevin Pulo
            Reporter:
            Allen K
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: