-
Type:
Bug
-
Resolution: Done
-
Priority:
Blocker - P1
-
None
-
Affects Version/s: 2.4.6
-
Component/s: Internal Code
-
None
-
Environment:ubuntu 13.04 running on AWS instance with data on an EBS on ext4.
-
Linux
-
None
-
0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
I have a replicaset with the primary running mongodb 2.2 and a secondary running 2.4.6 and an arbiter.
The secondary just crashed and cannot restart due to corruption.
Here is the error log:
Fri Nov 1 22:20:15.270 [repl prefetch worker] Assertion: 10334:BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
0xdddd81 0xd9f55b 0xd9fa9c 0x6ed13f 0x90bdd4 0xb304cd 0xb321c2 0xc23602 0xdab721 0xe26609 0x7f5288704f8e 0x7f5287a08e1d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd9f55b]
/usr/bin/mongod() [0xd9fa9c]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ed13f]
/usr/bin/mongod(_ZN5mongo7Helpers8findByIdERNS_6ClientEPKcNS_7BSONObjERS5_PbS7_+0xcd4) [0x90bdd4]
/usr/bin/mongod(_ZN5mongo19prefetchRecordPagesEPKcRKNS_7BSONObjE+0x44d) [0xb304cd]
/usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x6e2) [0xb321c2]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) [0xc23602]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
/usr/bin/mongod() [0xe26609]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f5288704f8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5287a08e1d]
Fri Nov 1 22:20:15.276 [repl writer worker 1] Assertion: 10334:BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO
0xdddd81 0xd9f55b 0xd9fa9c 0x6ed13f 0xa8f587 0xa924c7 0xa72449 0xc273d3 0xc26b18 0xdab721 0xe26609 0x7f5288704f8e 0x7f5287a08e1d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd9f55b]
/usr/bin/mongod() [0xd9fa9c]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ed13f]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2067) [0xa8f587]
/usr/bin/mongod(_ZN5mongo27updateObjectsForReplicationEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0xb7) [0xa924c7]
/usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0xb39) [0xa72449]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x713) [0xc273d3]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x48) [0xc26b18]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
/usr/bin/mongod() [0xe26609]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f5288704f8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5287a08e1d]
Fri Nov 1 22:20:15.282 [repl writer worker 1] ERROR: writer worker caught exception: BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO on: { ts: Timestamp 1383334273000|29, h: -7997199756652111606, v: 2, op: "u", ns: "webdoc_production.digest_mails", o2: { _id: ObjectId('5274056f1cbbe237b9000001') }, o: { $set: { sent_at: new Date(1383335279245) } } }
Fri Nov 1 22:20:15.282 [repl writer worker 1] Fatal Assertion 16360
0xdddd81 0xd9dc13 0xc26bfc 0xdab721 0xe26609 0x7f5288704f8e 0x7f5287a08e1d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd9dc13]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc26bfc]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
/usr/bin/mongod() [0xe26609]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f5288704f8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5287a08e1d]
Fri Nov 1 22:20:15.289 [repl writer worker 1]
***aborting after fassert() failure
Fri Nov 1 22:20:15.289 Got signal: 6 (Aborted).
Fri Nov 1 22:20:15.299 Backtrace:
0xdddd81 0x6d0d29 0x7f52879460b0 0x7f5287946037 0x7f5287949698 0xd9dc4e 0xc26bfc 0xdab721 0xe26609 0x7f5288704f8e 0x7f5287a08e1d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdddd81]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d29]
/lib/x86_64-linux-gnu/libc.so.6(+0x370b0) [0x7f52879460b0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f5287946037]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f5287949698]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xd9dc4e]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc26bfc]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdab721]
/usr/bin/mongod() [0xe26609]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7f8e) [0x7f5288704f8e]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5287a08e1d]