[SERVER-3135] mmap private failed with out of memory - Assertion: 13636:createPrivateMap failed Created: 24/May/11  Updated: 12/Jul/16  Resolved: 27/Jul/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: Sebastian Blask Assignee: Mathias Stearn
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS release 5.4 (Final) Kernel 2.6.18-194.17.1.el5


Operating System: Linux
Participants:

 Description   

Possibly related to SERVER-2597 but without crash.
On a 64bit system I start mongo 1.8.1 on existing data (30G) which was created by MongoDB 1.6.5 (which never caused problems):

/data/labs/bin/mongod --dbpath /data/labs/db --nohttpinterface --journal --port 5003

Everything I do end up in an exception, even in db shell:

> db.stats()
{
"assertion" : "createPrivateMap failed (look in log for error)",
"assertionCode" : 13636,
"errmsg" : "db assertion failure",
"ok" : 0
}
> show dbs
Tue May 24 15:47:16 uncaught exception: listDatabases failed:{
"errmsg" : "exception: createPrivateMap failed (look in log for error)",
"code" : 13636,
"ok" : 0
}

The exception taken from command line is:

Tue May 24 15:46:08 [initandlisten] MongoDB starting : pid=24245 port=5003 dbpath=/data/labs/db/ 64-bit
Tue May 24 15:46:08 [initandlisten] db version v1.8.1, pdfile version 4.5
Tue May 24 15:46:08 [initandlisten] git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04
Tue May 24 15:46:08 [initandlisten] build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue May 24 15:46:08 [initandlisten] journal dir=/data/labs/db/journal
Tue May 24 15:46:08 [initandlisten] recover : no journal files present, no recovery needed
Tue May 24 15:46:08 [initandlisten] waiting for connections on port 5003
Tue May 24 15:46:42 [initandlisten] connection accepted from 127.0.0.1:50990 #1
Tue May 24 15:46:48 [conn1] ERROR: mmap private failed with out of memory. (64 bit build)
Tue May 24 15:46:48 [conn1] Assertion: 13636:createPrivateMap failed (look in log for error)
0x55ece9 0x627520 0x627e8c 0x773c2a 0x7677bb 0x767e42 0x769c29 0x76a43b 0x762056 0x76286e 0x7db1cf 0x7dc741 0x642c25 0x6477c6 0x752225 0x757938 0x8a3b3e 0x8b6a40 0x3ed3406617 0x3ed2cd3c2d
/data/labs/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x55ece9]
/data/labs/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x60) [0x627520]
/data/labs/bin/mongod(_ZN5mongo8MongoMMF6createESsRyb+0x6c) [0x627e8c]
/data/labs/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x13a) [0x773c2a]
/data/labs/bin/mongod(_ZN5mongo8Database7getFileEiib+0x17b) [0x7677bb]
/data/labs/bin/mongod(_ZN5mongo8Database12openAllFilesEv+0x22) [0x767e42]
/data/labs/bin/mongod(_ZN5mongo8DatabaseC1EPKcRbRKSs+0x459) [0x769c29]
/data/labs/bin/mongod(_ZN5mongo14DatabaseHolder11getOrCreateERKSsS2_Rb+0x5bb) [0x76a43b]
/data/labs/bin/mongod(_ZN5mongo6Client7Context11_finishInitEb+0x516) [0x762056]
/data/labs/bin/mongod(_ZN5mongo6Client7ContextC1ERKSsSsPNS_9mongolockEb+0x7e) [0x76286e]
/data/labs/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x95f) [0x7db1cf]
/data/labs/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x831) [0x7dc741]
/data/labs/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x642c25]
/data/labs/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x2e86) [0x6477c6]
/data/labs/bin/mongod [0x752225]
/data/labs/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5b8) [0x757938]
/data/labs/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a3b3e]
/data/labs/bin/mongod(thread_proxy+0x80) [0x8b6a40]
/lib64/libpthread.so.0 [0x3ed3406617]
/lib64/libc.so.6(clone+0x6d) [0x3ed2cd3c2d]

More information:

free -ltm
total used free shared buffers cached
Mem: 16056 12479 3576 0 1 10205
Low: 16056 12479 3576
High: 0 0 0
-/+ buffers/cache: 2272 13783
Swap: 16387 1839 14547
Total: 32444 14319 18124

ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 136192
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 136192
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited



 Comments   
Comment by Eliot Horowitz (Inactive) [ 27/Jul/11 ]

See SERVER-3489

Comment by Alexander Azarov [ 27/Jul/11 ]

Ok, I looked at SERVER-3476 as well and tried to set overcommit_memory=0. This resolved the issue

Comment by Alexander Azarov [ 27/Jul/11 ]

It's 2

Comment by Mathias Stearn [ 26/Jul/11 ]

what is the value of /proc/sys/vm/overcommit_memory?

Comment by Alexander Azarov [ 26/Jul/11 ]

I have this same error.

  1. uname -a
    Linux mgmt 2.6.26-2-vserver-amd64 #1 SMP Tue Jan 25 06:09:17 UTC 2011 x86_64 GNU/Linux
  1. ulimit -v
    unlimited

I am trying to attach a new node to my ReplicaSet cluster. mongod throws these errors into the log continuously in the end of "sync". The log:

Wed Jul 27 00:49:48 [initandlisten] MongoDB starting : pid=27296 port=27017 dbpath=/var/lib/mongodb 64-bit
Wed Jul 27 00:49:48 [initandlisten] db version v1.8.2, pdfile version 4.5
Wed Jul 27 00:49:48 [initandlisten] git version: nogitversion
Wed Jul 27 00:49:48 [initandlisten] build sys info: Linux bobek-a0 2.6.37-2-amd64 #1 SMP Sun Feb 27
10:12:22 UTC 2011 x86_64 BOOST_LIB_VERSION=1_46_1
Wed Jul 27 00:49:48 [initandlisten] journal dir=/var/lib/mongodb/journal
.....

Wed Jul 27 01:55:28 [replica set sync] external sort used : 13 files in 113 secs
698100/12189696 5%
2011300/12189696 16%
Wed Jul 27 01:55:55 [replica set sync] ERROR: mmap private failed with out of memory. (64 bit build)
Wed Jul 27 01:55:55 [replica set sync] Assertion: 13636:createPrivateMap failed (look in log for err
or)
0x566088 0x60be91 0x60bf07 0x70217f 0x6f12d3 0x6f14ef 0x6f1591 0x6f1772 0x6fd832 0x64fd91 0x64fde4 0
x651a60 0x6fa8cb 0x6fcc83 0x6febf8 0x6c2498 0x6c469a 0x6c4f6a 0x6b242e 0x6b30b5
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x138) [0x566088]
/usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x71) [0x60be91]
/usr/bin/mongod(_ZN5mongo8MongoMMF6createESsRyb+0x67) [0x60bf07]
/usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x12f) [0x70217f]
/usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xc3) [0x6f12d3]
/usr/bin/mongod(_ZN5mongo8Database8addAFileEib+0x2f) [0x6f14ef]
/usr/bin/mongod(_ZN5mongo8Database12suitableFileEib+0x61) [0x6f1591]
/usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcib+0x72) [0x6f1772]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x1562) [0x6fd832]
/usr/bin/mongod(_ZN5mongo11BtreeBucket9addBucketERKNS_12IndexDetailsE+0x61) [0x64fd91]
/usr/bin/mongod(_ZN5mongo12BtreeBuilder9newBucketEv+0x14) [0x64fde4]
/usr/bin/mongod(_ZN5mongo12BtreeBuilder6addKeyERNS_7BSONObjENS_7DiskLocE+0x210) [0x651a60]
/usr/bin/mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xd8b) [0x6fa8cb]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x9b3) [0x6fcc83]
/usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x48) [0x6febf8]
/usr/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x4c8) [0x6c2498]
/usr/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x135a) [0x6c469a]
/usr/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3a) [0x6c4f6a]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0xb5e) [0x6b242e]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl17syncDoInitialSyncEv+0x15) [0x6b30b5]
Wed Jul 27 01:55:55 [replica set sync] ERROR: mmap private failed with out of memory. (64 bit build)
Wed Jul 27 01:55:55 [replica set sync] Assertion: 13636:createPrivateMap failed (look in log for error)
0x566088 0x60be91 0x60bf07 0x70217f 0x6f12d3 0x6f156e 0x6f1772 0x6fd832 0x64fd91 0x64fde4 0x651a60 0x6fa8cb 0x6fcc83 0x6febf8 0x6c2498 0x6c469a 0x6c4f6a 0x6b242e 0x6b30b5 0x6aeb58
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x138) [0x566088]
/usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x71) [0x60be91]
/usr/bin/mongod(_ZN5mongo8MongoMMF6createESsRyb+0x67) [0x60bf07]
/usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x12f) [0x70217f]
/usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xc3) [0x6f12d3]
/usr/bin/mongod(_ZN5mongo8Database12suitableFileEib+0x3e) [0x6f156e]
/usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcib+0x72) [0x6f1772]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x1562) [0x6fd832]
/usr/bin/mongod(_ZN5mongo11BtreeBucket9addBucketERKNS_12IndexDetailsE+0x61) [0x64fd91]
/usr/bin/mongod(_ZN5mongo12BtreeBuilder9newBucketEv+0x14) [0x64fde4]
/usr/bin/mongod(_ZN5mongo12BtreeBuilder6addKeyERNS_7BSONObjENS_7DiskLocE+0x210) [0x651a60]
/usr/bin/mongod(_ZN5mongo14fastBuildIndexEPKcPNS_16NamespaceDetailsERNS_12IndexDetailsEi+0xd8b) [0x6fa8cb]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibRKNS_11BSONElementEb+0x9b3) [0x6fcc83]
/usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEb+0x48) [0x6febf8]
/usr/bin/mongod(_ZN5mongo6Cloner4copyEPKcS2_bbbbNS_5QueryE+0x4c8) [0x6c2498]
/usr/bin/mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x135a) [0x6c469a]
/usr/bin/mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x3a) [0x6c4f6a]
/usr/bin/mongod(_ZN5mongo11ReplSetImpl18_syncDoInitialSyncEv+0xb5e) [0x6b242e]

and so on

Generated at Thu Feb 08 03:02:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.