[SERVER-10189] Server crash under load Created: 12/Jul/13  Updated: 10/Dec/14  Resolved: 12/Jul/13

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Dharshan Rangegowda Assignee: Andy Schwerin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Looks like a lack of memory. Either way I would expect it to slow down and not crash.

Fri Jul 12 05:21:59.717 [conn843] authenticate db: local

{ authenticate: 1, nonce: "237b181d39d143ae", user: "__system", key: "3597fb4d02ab28d832f8d 993" }

Fri Jul 12 05:22:07.589 [conn782] serverStatus was very slow:

{ after basic: 0, after asserts: 0, after backgroundFlushing: 0, after connections: 0, rsors: 0, after dur: 0, after extra_info: 0, after globalLock: 0, after indexCounters: 0, after locks: 0, after network: 0, after opcounters: 0, afte tersRepl: 0, after recordStats: 4010, after repl: 4020, at end: 4020 }

Fri Jul 12 05:22:07.591 [conn782] command admin.$cmd command:

{ serverStatus: 1 }

ntoreturn:1 keyUpdates:0 locks(micros) r:30 reslen:4014 5037ms
Fri Jul 12 05:22:07.652 [conn782] command denied:

{ writeBacksQueued: 1 }

Fri Jul 12 05:22:11.901 [journal] Assertion: 13524:out of memory AlignedBuilder
0xdc7f71 0xd8963b 0xd8760c 0xd87639 0x91f55e 0x91f922 0x914b11 0x916759 0x916b1b 0xe10879 0x7fc0107fb851 0x7fc00fb9e11d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdc7f71]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd8963b]
/usr/bin/mongod() [0xd8760c]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder8_reallocEjj+0x29) [0xd87639]
/usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x4e) [0x91f55e]
/usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x91f922]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0x914b11]
/usr/bin/mongod() [0x916759]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x916b1b]
/usr/bin/mongod() [0xe10879]
/lib64/libpthread.so.0(+0x7851) [0x7fc0107fb851]
/lib64/libc.so.6(clone+0x6d) [0x7fc00fb9e11d]
Fri Jul 12 05:22:12.045 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Fri Jul 12 05:22:12.045 dur1
Fri Jul 12 05:22:12.049 Got signal: 6 (Aborted).

Fri Jul 12 05:22:12.059 Backtrace:
0xdc7f71 0x6ce459 0x7fc00fae8920 0x7fc00fae88a5 0x7fc00faea085 0x9e3ee7 0x914f3f 0x916759 0x916b1b 0xe10879 0x7fc0107fb851 0x7fc00fb9e11d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdc7f71]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6ce459]
/lib64/libc.so.6(+0x32920) [0x7fc00fae8920]
/lib64/libc.so.6(gsignal+0x35) [0x7fc00fae88a5]
/lib64/libc.so.6(abort+0x175) [0x7fc00faea085]
/usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9e3ee7]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0x914f3f]
/usr/bin/mongod() [0x916759]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x916b1b]
/usr/bin/mongod() [0xe10879]
/lib64/libpthread.so.0(+0x7851) [0x7fc0107fb851]
/lib64/libc.so.6(clone+0x6d) [0x7fc00fb9e11d]



 Comments   
Comment by Andy Schwerin [ 12/Jul/13 ]

Your process has apparently exhausted free memory and swap space. Under this kind of memory pressure, you're pretty likely to get killed by the kernel OOM-killer eventually, too. To keep write latency low, MongoDB requires very large amounts of RAM for buffering disk operations.

Comment by Dharshan Rangegowda [ 12/Jul/13 ]

2.4.1

Comment by Andy Schwerin [ 12/Jul/13 ]

And what version of MongoD?

Comment by Dharshan Rangegowda [ 12/Jul/13 ]

It's an Amazon linux micro instance.

Comment by Andy Schwerin [ 12/Jul/13 ]

Version? OS? 64- or 32-bit?

Generated at Thu Feb 08 03:22:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.