[SERVER-2597] createPrivateMap fails Created: 20/Feb/11  Updated: 12/Jul/16  Resolved: 09/Mar/11

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 1.7.5, 1.7.6
Fix Version/s: 1.8.0-rc2

Type: Bug Priority: Major - P3
Reporter: Justin Smestad Assignee: Mathias Stearn
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 10.04 LTS 64-bit build


Operating System: Linux
Participants:

 Description   

Testing 1.7.x build on ubuntu the process quits with the following error on launch:

Sun Feb 20 20:55:59 [initandlisten] MongoDB starting : pid=15317 port=27017 dbpath=/data/mongodb master=1 64-bit
Sun Feb 20 20:55:59 [initandlisten] db version v1.8.0-rc0-pre-, pdfile version 4.5
Sun Feb 20 20:55:59 [initandlisten] git version: 9a6f2dcf91c2025f7549c875e78e4a203b26adcf
Sun Feb 20 20:55:59 [initandlisten] sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Sun Feb 20 20:55:59 [initandlisten] journal dir=/data/mongodb/journal
Sun Feb 20 20:55:59 [initandlisten] recover : no journal files present, no recovery needed
Sun Feb 20 20:55:59 [initandlisten] waiting for connections on port 27017
Sun Feb 20 20:55:59 [initandlisten] ERROR: mmap private failed with out of memory. (64 bit build)
Sun Feb 20 20:55:59 [initandlisten] Assertion: 13636:createPrivateMap failed (look in log for error)
0x560b39 0x626ac0 0x62785b 0x769f5d 0x763b88 0x764282 0x76476c 0x764fff 0x75ce8f 0x75d82e 0x742beb 0x6924d3 0x89ed90 0x89fbbc 0x8a02c2 0x8a6283 0x7f2c967b7c4d 0x4e03d9
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x560b39]
/usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x60) [0x626ac0]
/usr/bin/mongod(_ZN5mongo8MongoMMF6createESsRyb+0xdb) [0x62785b]
/usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x12d) [0x769f5d]
/usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0x178) [0x763b88]
/usr/bin/mongod(_ZN5mongo8Database12openAllFilesEv+0x22) [0x764282]
/usr/bin/mongod(_ZN5mongo8DatabaseC1EPKcRbRKSs+0x44c) [0x76476c]
/usr/bin/mongod(_ZN5mongo14DatabaseHolder11getOrCreateERKSsS2_Rb+0x5df) [0x764fff]
/usr/bin/mongod(_ZN5mongo6Client7Context11_finishInitEb+0x38f) [0x75ce8f]
/usr/bin/mongod(_ZN5mongo6Client7ContextC1ERKSsSsPNS_9mongolockEb+0x7e) [0x75d82e]
/usr/bin/mongod(_ZN5mongo7Helpers12getSingletonEPKcRNS_7BSONObjE+0x6b) [0x742beb]
/usr/bin/mongod(_ZN5mongo16startReplicationEv+0x2b3) [0x6924d3]
/usr/bin/mongod(_ZN5mongo6listenEi+0xd0) [0x89ed90]
/usr/bin/mongod(_ZN5mongo14_initAndListenEiPKc+0x8dc) [0x89fbbc]
/usr/bin/mongod(_ZN5mongo13initAndListenEiPKc+0x22) [0x8a02c2]
/usr/bin/mongod(main+0x5c83) [0x8a6283]
/lib/libc.so.6(__libc_start_main+0xfd) [0x7f2c967b7c4d]
/usr/bin/mongod(__gxx_personality_v0+0x3b9) [0x4e03d9]
Sun Feb 20 20:55:59 [initandlisten] exception in initAndListen std::exception: createPrivateMap failed (look in log for error), terminating
Sun Feb 20 20:55:59 dbexit:
Sun Feb 20 20:55:59 [initandlisten] shutdown: going to close listening sockets...
Sun Feb 20 20:55:59 [initandlisten] shutdown: going to flush diaglog...
Sun Feb 20 20:55:59 [initandlisten] shutdown: going to close sockets...
Sun Feb 20 20:55:59 [initandlisten] shutdown: waiting for fs preallocator...
Sun Feb 20 20:55:59 [initandlisten] shutdown: lock for final commit...
Sun Feb 20 20:55:59 [initandlisten] shutdown: final commit...
Sun Feb 20 20:55:59 [initandlisten] shutdown: closing all files...
Sun Feb 20 20:55:59 closeAllFiles() finished
Sun Feb 20 20:55:59 [initandlisten] shutdown: journalCleanup...
Sun Feb 20 20:55:59 [initandlisten] removeJournalFiles
Sun Feb 20 20:55:59 [initandlisten] shutdown: removing fs lock...
Sun Feb 20 20:55:59 dbexit: really exiting now

Not sure if this is related to --dur or not.



 Comments   
Comment by Dwight Merriman [ 08/Mar/11 ]

mathias can you work with tony on this; tony can reproduce.

Comment by Tony Hannan [ 08/Mar/11 ]

My mongod log of failure on insert:

Tue Mar 8 15:58:45 [conn1] ERROR: mmap private failed with out of memory. (64 bit build)
Tue Mar 8 15:58:45 [conn1] Assertion: 13636:createPrivateMap failed (look in log for error)
0x5620b9 0x629dc0 0x62a72c 0x7719ea 0x768bfb 0x769282 0x76b069 0x76b87b 0x76330f 0x763cae 0x755a30 0x75b66c 0x8a893e 0x8b8d80 0x7fa6ddea7fc7 0x7fa6dd4775ad
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x5620b9]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x60) [0x629dc0]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8MongoMMF6createESsRyb+0x6c) [0x62a72c]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x13a) [0x7719ea]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8Database7getFileEiib+0x17b) [0x768bfb]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8Database12openAllFilesEv+0x22) [0x769282]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo8DatabaseC1EPKcRbRKSs+0x459) [0x76b069]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo14DatabaseHolder11getOrCreateERKSsS2_Rb+0x5bb) [0x76b87b]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo6Client7Context11_finishInitEb+0x38f) [0x76330f]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo6Client7ContextC1ERKSsSsPNS_9mongolockEb+0x7e) [0x763cae]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo14receivedInsertERNS_7MessageERNS_5CurOpE+0x220) [0x755a30]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x18ec) [0x75b66c]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x8a893e]
mongodb-linux-x86_64-v1.8-2011-03-08/bin/mongod(thread_proxy+0x80) [0x8b8d80]
/lib/libpthread.so.0 [0x7fa6ddea7fc7]
/lib/libc.so.6(clone+0x6d) [0x7fa6dd4775ad]
Tue Mar 8 15:58:45 [conn1] insert test.foo exception 13636 createPrivateMap failed (look in log for error) 0ms

Comment by Justin Smestad [ 20/Feb/11 ]

I had the following database sizes:

> show dbs;
admin 0.203125GB
local 2.2021484375GB
mongo_stats 0.453125GB

After deleting local.main.$oplog and running db.repairDatabase(); the size went down to:

> show dbs;
admin 0.203125GB
local 0.203125GB
mongo_stats 0.453125GB

After doing this the server will startup properly.

Will the database operate properly as additional data is entered or will it crash again? I'm assuming this is some edge case when launching the first time with --dur enabled on an existing data set, correct?

Comment by Scott Hernandez (Inactive) [ 20/Feb/11 ]

Posted on IRC:

total used free shared buffers cached
Mem: 995 577 417 0 74 433
Low: 995 577 417
High: 0 0 0
-/+ buffers/cache: 69 925
Swap: 255 7 248
Total: 1251 584 666

Comment by Eliot Horowitz (Inactive) [ 20/Feb/11 ]

How much physical + swap space do you have on that box?
Were you starting with an empty db or existing data?
If existing data, how much?

Comment by Scott Hernandez (Inactive) [ 20/Feb/11 ]

Justin, can you post the output of "free -ltm" before you start it up with the nightly?

Comment by Justin Smestad [ 20/Feb/11 ]

This does seem to only happen with -dur turned on. It happens with 1.7.5, 1.7.6 and 1.8.0-rc-pre-

Generated at Thu Feb 08 03:00:30 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.