[SERVER-17559] Replica set member down after out-of-memory condition Created: 12/Mar/15  Updated: 12/Mar/15  Resolved: 12/Mar/15

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 2.4.12
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: sampat Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Thu Mar 12 15:11:29.984 [initandlisten] MongoDB starting : pid=20213 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=ubuntu2
Thu Mar 12 15:11:29.984 [initandlisten] db version v2.4.12
Thu Mar 12 15:11:29.984 [initandlisten] git version: 09917767b116f4ff1c0eadda1e8bc5db30828500
Thu Mar 12 15:11:29.984 [initandlisten] build info: Linux ip-10-142-184-243 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Mar 12 15:11:29.984 [initandlisten] allocator: tcmalloc
Thu Mar 12 15:11:29.984 [initandlisten] options: { dbpath: "/var/lib/mongodb/", keyFile: "/data/keyfile", replSet: "prodRepl" }
Thu Mar 12 15:11:30.065 [initandlisten] journal dir=/var/lib/mongodb/journal
Thu Mar 12 15:11:30.099 [initandlisten] recover begin
Thu Mar 12 15:11:30.099 [initandlisten] info no lsn file in journal/ directory
Thu Mar 12 15:11:30.100 [initandlisten] recover lsn: 0
Thu Mar 12 15:11:30.100 [initandlisten] recover /var/lib/mongodb/journal/j._0
Thu Mar 12 15:11:30.112 [initandlisten] recover cleaning up
Thu Mar 12 15:11:30.112 [initandlisten] removeJournalFiles
Thu Mar 12 15:11:30.115 [initandlisten] recover done
Thu Mar 12 15:11:31.378 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0  reslen:75 340ms
Thu Mar 12 15:11:31.386 [journal] Assertion: 13524:out of memory AlignedBuilder
Thu Mar 12 15:11:31.391 [websvr] admin web console waiting for connections on port 28017
Thu Mar 12 15:11:31.440 [initandlisten] waiting for connections on port 27017
0xdea5b1 0xdabd7b 0xda9d4c 0xda9fe0 0x930601 0x9307f2 0x925961 0x927c39 0x92800b 0xe33239 0x7f4ba5790182 0x7f4ba4a9547d
Thu Mar 12 15:11:31.609 [rsStart] replSet I am ubuntu2:27017
Thu Mar 12 15:11:31.613 [rsHealthPoll] replset info ubuntu3:27017 thinks that we are down
Thu Mar 12 15:11:31.614 [rsHealthPoll] replSet member ubuntu3:27017 is up
Thu Mar 12 15:11:31.614 [rsHealthPoll] replSet member ubuntu3:27017 is now in state PRIMARY
Thu Mar 12 15:11:31.618 [rsStart] replSet STARTUP2
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdea5b1]
 mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xdabd7b]
 mongod() [0xda9d4c]
 mongod(_ZN5mongo14AlignedBuilderC1Ej+0x10) [0xda9fe0]
 mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x221) [0x930601]
 mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x9307f2]
 mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0x925961]
 mongod() [0x927c39]
 mongod(_ZN5mongo3dur9durThreadEv+0x2fb) [0x92800b]
 mongod() [0xe33239]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f4ba5790182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4ba4a9547d]
Thu Mar 12 15:11:31.649 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Thu Mar 12 15:11:31.649 dur1
Thu Mar 12 15:11:31.650 Got signal: 6 (Aborted).
 
Thu Mar 12 15:11:31.659 Backtrace:
0xdea5b1 0x6d0f29 0x7f4ba49d1d40 0x7f4ba49d1cc9 0x7f4ba49d50d8 0x9f6947 0x925d8f 0x927c39 0x92800b 0xe33239 0x7f4ba5790182 0x7f4ba4a9547d
 mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdea5b1]
 mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0f29]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f4ba49d1d40]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f4ba49d1cc9]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f4ba49d50d8]
 mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9f6947]
 mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0x925d8f]
 mongod() [0x927c39]
 mongod(_ZN5mongo3dur9durThreadEv+0x2fb) [0x92800b]
 mongod() [0xe33239]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f4ba5790182]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f4ba4a9547d]



 Comments   
Comment by Ramon Fernandez Marina [ 12/Mar/15 ]

Hi sampat, you're right about the cause. If you check your system logs you'll probably also find a message like:

Killed process: mongod

from the OOM killer. To keep write latency low, MongoDB requires very large amounts of RAM for buffering disk operations.

Regards,
Ramón.

Comment by sampat [ 12/Mar/15 ]

Solved the issue: it was due to low memory

Generated at Thu Feb 08 03:44:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.