Loading...

Type: Bug
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 2.4.5
Component/s: Replication
Labels:
None
Environment:
Ubuntu 12.04.2 LTS
VPS, 4GB of RAM
Replica set: 1 Arbiter, 3 secondaries, 1 primary
No sharding

Operating System:
ALL

We had this entry in our mongodb.log on the primary:

Fri Oct  4 18:57:37.062 [rsHealthPoll] DBClientCursor::init call() failed

Then the heartbeat started failing, and then it went down completely:

Fri Oct  4 18:57:37.112 [rsHealthPoll] replset info mongo.1.prod.X.com:27017 heartbeat failed, retrying
Fri Oct  4 18:57:37.117 [rsHealthPoll] replSet info mongo.1.prod.X.com:27017 is down (or slow to respond): 
Fri Oct  4 18:57:37.117 [rsHealthPoll] replSet member mongo.1.prod.X.com:27017 is now in state DOWN

On this particular secondary, we noticed this in the logs and then the Mongo service entirely died:

Fri Oct  4 18:57:35.800 [repl writer worker 1] ERROR: mmap private failed with out of memory. (64 bit build)
Fri Oct  4 18:57:35.823 [repl writer worker 1] Assertion: 13636:file /var/lib/mongodb/statistics_prod.5 open/create failed in createPrivateMap (look in log for more information)
0xdd9e31 0xd9b7ab 0xd9bcec 0xa5695b 0xa572ba 0xab66d1 0x8cf00d 0x8cf518 0x8cf5ff 0x8cf89e 0xab0a30 0xab0c8f 0xac25e8 0xac318f 0xa8dcaa 0xa8fd77 0xa6f80c 0xc23c03 0xc23348 0xda7971 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd9b7ab]
 /usr/bin/mongod() [0xd9bcec]
 /usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x1fb) [0xa5695b]
 /usr/bin/mongod(_ZN5mongo8MongoMMF6createERKSsRyb+0x5a) [0xa572ba]
 /usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x141) [0xab66d1]
 /usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xbd) [0x8cf00d]
 /usr/bin/mongod(_ZN5mongo8Database8addAFileEib+0x38) [0x8cf518]
 /usr/bin/mongod(_ZN5mongo8Database12suitableFileEPKcibb+0xaf) [0x8cf5ff]
 /usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcibb+0x9e) [0x8cf89e]
 /usr/bin/mongod(_ZN5mongo10outOfSpaceEPKcPNS_16NamespaceDetailsEib+0xb0) [0xab0a30]
 /usr/bin/mongod(_ZN5mongo26allocateSpaceForANewRecordEPKcPNS_16NamespaceDetailsEib+0x4f) [0xab0c8f]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x1998) [0xac25e8]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac318f]
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2eda) [0xa8dcaa]
 /usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fd77]
 /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x65c) [0xa6f80c]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc23c03]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc23348]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
Fri Oct  4 18:57:35.971 [repl writer worker 1] ERROR: writer worker caught exception: file /var/lib/mongodb/statistics_prod.5 open/create failed in createPrivateMap (look in log for more information) on: { ts: Timestamp 1380927455000|5, h: -5106974538304874419, v: 2, op: "i", ns: (removed for security) } }
Fri Oct  4 18:57:35.971 [repl writer worker 1]   Fatal Assertion 16360
0xdd9e31 0xd99e63 0xc2342c 0xda7971 0xe225d9 0x7f1f78df0e9a 0x7f1f78103ccd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd99e63]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /usr/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1f78df0e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1f78103ccd]
Fri Oct  4 18:57:35.975 [repl writer worker 1] 

***aborting after fassert() failure


Fri Oct  4 18:57:35.975 Got signal: 6 (Aborted).

Fri Oct  4 18:57:36.002 Backtrace:
0xdd9e31 0x6d0d09 0x7f1f780464a0 0x7f1f78046425 0x7f1f78049b8b 0xd99e9e 0xc2342c 0xda7971 0xe225d9 0x7f1f78df0e9a 0x7f1f78103ccd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d09]
 /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f1f780464a0]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f1f78046425]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f1f78049b8b]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd99e9e]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc2342c]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xda7971]
 /usr/bin/mongod() [0xe225d9]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f1f78df0e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f1f78103ccd]

So at this point we have one secondary down. The primary logs continue for about the next five minutes saying it's still down. But then the primary encounters some serious, similar issues of its own.

Fri Oct  4 19:02:39.032 [conn198845] ERROR: mmap private failed with out of memory. (64 bit build)
Fri Oct  4 19:02:39.044 [conn198845] Assertion: 13636:file /var/lib/mongodb/statistics_prod.5 open/create failed in createPrivateMap (look in log for more information)
0xdd9e31 0xd9b7ab 0xd9bcec 0xa5695b 0xa572ba 0xab66d1 0x8cf00d 0x8cf518 0x8cf5ff 0x8cf89e 0xab0a30 0xab0c8f 0xac12e5 0xac318f 0xa8ca47 0xa8fe67 0x9f2ff8 0x9f8588 0x6e8b68 0xdc659e 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd9b7ab]
 /usr/bin/mongod() [0xd9bcec]
 /usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x1fb) [0xa5695b]
 /usr/bin/mongod(_ZN5mongo8MongoMMF6createERKSsRyb+0x5a) [0xa572ba]
 /usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x141) [0xab66d1]
 /usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xbd) [0x8cf00d]
 /usr/bin/mongod(_ZN5mongo8Database8addAFileEib+0x38) [0x8cf518]
 /usr/bin/mongod(_ZN5mongo8Database12suitableFileEPKcibb+0xaf) [0x8cf5ff]
 /usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcibb+0x9e) [0x8cf89e]
 /usr/bin/mongod(_ZN5mongo10outOfSpaceEPKcPNS_16NamespaceDetailsEib+0xb0) [0xab0a30]
 /usr/bin/mongod(_ZN5mongo26allocateSpaceForANewRecordEPKcPNS_16NamespaceDetailsEib+0x4f) [0xab0c8f]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x695) [0xac12e5]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac318f]
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x1c77) [0xa8ca47]
 /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fe67]
 /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4d8) [0x9f2ff8]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xac8) [0x9f8588]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e8b68]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdc659e]
Fri Oct  4 19:02:39.761 [conn198845] ERROR: mmap private failed with out of memory. (64 bit build)
Fri Oct  4 19:02:39.761 [conn198845] Assertion: 13636:file /var/lib/mongodb/statistics_prod.5 open/create failed in createPrivateMap (look in log for more information)
0xdd9e31 0xd9b7ab 0xd9bcec 0xa5695b 0xa572ba 0xab66d1 0x8cf00d 0x8cf599 0x8cf89e 0xab0a30 0xab0c8f 0xac12e5 0xac318f 0xa8ca47 0xa8fe67 0x9f2ff8 0x9f8588 0x6e8b68 0xdc659e 0x7f27e81fce9a 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd9b7ab]
 /usr/bin/mongod() [0xd9bcec]
 /usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x1fb) [0xa5695b]
 /usr/bin/mongod(_ZN5mongo8MongoMMF6createERKSsRyb+0x5a) [0xa572ba]
 /usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x141) [0xab66d1]
 /usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xbd) [0x8cf00d]
 /usr/bin/mongod(_ZN5mongo8Database12suitableFileEPKcibb+0x49) [0x8cf599]
 /usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcibb+0x9e) [0x8cf89e]
 /usr/bin/mongod(_ZN5mongo10outOfSpaceEPKcPNS_16NamespaceDetailsEib+0xb0) [0xab0a30]
 /usr/bin/mongod(_ZN5mongo26allocateSpaceForANewRecordEPKcPNS_16NamespaceDetailsEib+0x4f) [0xab0c8f]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x695) [0xac12e5]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac318f]
 /usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x1c77) [0xa8ca47]
 /usr/bin/mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa8fe67]
 /usr/bin/mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x4d8) [0x9f2ff8]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xac8) [0x9f8588]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x6e8b68]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x42e) [0xdc659e]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f27e81fce9a]

This continues for 2.5 hours until I manually step it down. Then about 2 seconds later, it finally dies completely

Fri Oct  4 21:30:24.123 [repl writer worker 1] 

***aborting after fassert() failure


Fri Oct  4 21:30:24.124 Got signal: 6 (Aborted).

related to

SERVER-1121 Crashes reported on OpenVZ

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates