|
I am pretty sure this is a memory problem. We have a 4GB+4GBswap system and we will try resolving by increasing to 8GB+8GBswap.
This has happened to us a couple times and results in needing to rebuild the node. Both times, it wiped out 2 of 3 nodes in the cluster. This leads to the remaining server downgrading itself from primary to secondary. I am worried about what would happen if this happened on all 3 nodes at the same time.
Is there any way to make Mongo more resilient about these problems and fail more gracefully.
This is what we get in the mongo.log file:
Thu Mar 21 15:14:01 [journal] warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f38ad18a8ca 0x7f38ac53db6d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73]
/usr/bin/mongod() [0x753da5]
/usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/libpthread.so.0(+0x68ca) [0x7f38ad18a8ca]
/lib/libc.so.6(clone+0x6d) [0x7f38ac53db6d]
Thu Mar 21 15:14:02 [conn94583] end connection 10.1.7.11:50309 (7 connections now open)
Thu Mar 21 15:14:02 [initandlisten] connection accepted from 10.1.7.11:50314 #94585 (8 connections now open)
Thu Mar 21 15:14:10 [conn94584] end connection 10.1.7.13:43214 (7 connections now open)
Thu Mar 21 15:14:10 [initandlisten] connection accepted from 10.1.7.13:43216 #94586 (8 connections now open)
Thu Mar 21 15:14:12 [journal] warning assertion failure a <= 256*1024*1024 src/mongo/util/alignedbuilder.cpp 90
0xade6e1 0x802c5a 0x77dc73 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f38ad18a8ca 0x7f38ac53db6d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo9wassertedEPKcS1_j+0x11a) [0x802c5a]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x63) [0x77dc73]
/usr/bin/mongod() [0x753da5]
/usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/libpthread.so.0(+0x68ca) [0x7f38ad18a8ca]
/lib/libc.so.6(clone+0x6d) [0x7f38ac53db6d]
Thu Mar 21 15:14:12 [journal] Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 91
0xade6e1 0x803dfd 0x77dc8d 0x753da5 0x7540b4 0xa09950 0xa0a779 0xa0ae24 0x7c3659 0x7f38ad18a8ca 0x7f38ac53db6d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0xfd) [0x803dfd]
/usr/bin/mongod(_ZN5mongo14AlignedBuilder14growReallocateEj+0x7d) [0x77dc8d]
/usr/bin/mongod() [0x753da5]
/usr/bin/mongod(_ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE+0x214) [0x7540b4]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0xa0) [0xa09950]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/libpthread.so.0(+0x68ca) [0x7f38ad18a8ca]
/lib/libc.so.6(clone+0x6d) [0x7f38ac53db6d]
Thu Mar 21 15:14:13 [journal] dbexception in groupCommitLL causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:91
Thu Mar 21 15:14:13 dur1
Thu Mar 21 15:14:13 Got signal: 6 (Aborted).
Thu Mar 21 15:14:14 Backtrace:
0xade6e1 0x5582d9 0x7f38ac4a0230 0x7f38ac4a01b5 0x7f38ac4a2fc0 0xb503f7 0xa09e1f 0xa0a779 0xa0ae24 0x7c3659 0x7f38ad18a8ca 0x7f38ac53db6d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
/lib/libc.so.6(+0x32230) [0x7f38ac4a0230]
/lib/libc.so.6(gsignal+0x35) [0x7f38ac4a01b5]
/lib/libc.so.6(abort+0x180) [0x7f38ac4a2fc0]
/usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0xb503f7]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0xa09e1f]
/usr/bin/mongod() [0xa0a779]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x364) [0xa0ae24]
/usr/bin/mongod() [0x7c3659]
/lib/libpthread.so.0(+0x68ca) [0x7f38ad18a8ca]
/lib/libc.so.6(clone+0x6d) [0x7f38ac53db6d]
|