Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical - P2
-
None
-
2.4.7
-
None
-
ALL
-
Description
Three member replica set, running version 2.4.7. Secondary died with an assert failure on invalid document size (Size must be between 0 and 16793600(16MB)). Two attempts to restart the secondary failed with the same error, on the same document. No issues at all with other secondary or primary. Ended up creating new storage volume and performing a resync to bring it back online.
I've included the info from the mongodb.log below. The document that had the invalid document size had just been inserted into the primary and replicated to the secondaries. Document looks fine on the primary and the surviving secondary. Actual document size is about 3kb.
Log info:
Wed Mar 19 01:05:55.184 [repl prefetch worker] Assertion: 10334:BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x7fc19b 0x7fc285 0x7fc285 0x7fc285 0x7fc285 0x7fc4c1 0x9d6aca 0x9dd34f 0xb329c4 0xb33865 0xc254b2 0xdada91
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
/usr/bin/mongod() [0xda1dcc]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]
/usr/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]
/usr/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]
/usr/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x6b) [0x7fc19b]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13twoStepInsertENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x1a1) [0x7fc4c1]
/usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0xda) [0x9d6aca]
/usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) [0x9dd34f]
/usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) [0xb329c4]
/usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) [0xb33865]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) [0xc254b2]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
Wed Mar 19 01:05:55.333 [repl writer worker 1] Assertion: 10334:BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x7fc19b 0x7fc285 0x7fc285 0x7fc285 0x7fc285 0x7fc4c1 0x9d6aca 0x9dd34f 0x9df975 0xac5b7d 0xac6e7f 0xa9198a
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
/usr/bin/mongod() [0xda1dcc]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]
/usr/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]
/usr/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]
/usr/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x6b) [0x7fc19b]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13twoStepInsertENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x1a1) [0x7fc4c1]
/usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0xda) [0x9d6aca]
/usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) [0x9dd34f]
/usr/bin/mongod(_ZN5mongo24indexRecordUsingTwoStepsEPKcPNS_16NamespaceDetailsENS_7BSONObjENS_7DiskLocEb+0x175) [0x9df975]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x123d) [0xac5b7d]
/usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac6e7f]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2eda) [0xa9198a]
Wed Mar 19 01:05:55.345 [repl writer worker 1] ERROR: writer worker caught exception: BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111 on: { ts: Timestamp 1395191155000|19, h: 8957154836310535201, v: 2, op: "i", ns:
<<<<Document Data Removed - size was about 3kb >>>>
Wed Mar 19 01:05:55.345 [repl writer worker 1] Fatal Assertion 16360
0xde0151 0xd9ff43 0xc28aac 0xdada91 0xe289d9 0x7f9c4d405e9a 0x7f9c4c7183fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd9ff43]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28aac]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
/usr/bin/mongod() [0xe289d9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9c4d405e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9c4c7183fd]
Wed Mar 19 01:05:55.350 [repl writer worker 1]
***aborting after fassert() failure