[SERVER-13279] Replica Set Secondary crashes with invalid size error Size must be between 0 and 16793600(16MB) Created: 19/Mar/14  Updated: 07/Aug/14  Resolved: 21/May/14

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 2.4.7
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: John Petrone Assignee: Bruce Lucas (Inactive)
Resolution: Cannot Reproduce Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

Unclear, although once it occurred could not bring mongod back up as it failed the same assertion on the same document every time.

Participants:

 Description   

Three member replica set, running version 2.4.7. Secondary died with an assert failure on invalid document size (Size must be between 0 and 16793600(16MB)). Two attempts to restart the secondary failed with the same error, on the same document. No issues at all with other secondary or primary. Ended up creating new storage volume and performing a resync to bring it back online.

I've included the info from the mongodb.log below. The document that had the invalid document size had just been inserted into the primary and replicated to the secondaries. Document looks fine on the primary and the surviving secondary. Actual document size is about 3kb.

Log info:

Wed Mar 19 01:05:55.184 [repl prefetch worker] Assertion: 10334:BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x7fc19b 0x7fc285 0x7fc285 0x7fc285 0x7fc285 0x7fc4c1 0x9d6aca 0x9dd34f 0xb329c4 0xb33865 0xc254b2 0xdada91
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
/usr/bin/mongod() [0xda1dcc]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]
/usr/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]
/usr/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]
/usr/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x6b) [0x7fc19b]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13twoStepInsertENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x1a1) [0x7fc4c1]
/usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0xda) [0x9d6aca]
/usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) [0x9dd34f]
/usr/bin/mongod(_ZN5mongo18prefetchIndexPagesEPNS_16NamespaceDetailsERKNS_7BSONObjE+0x724) [0xb329c4]
/usr/bin/mongod(_ZN5mongo28prefetchPagesForReplicatedOpERKNS_7BSONObjE+0x605) [0xb33865]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail10prefetchOpERKNS_7BSONObjE+0x202) [0xc254b2]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
Wed Mar 19 01:05:55.333 [repl writer worker 1] Assertion: 10334:BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111
0xde0151 0xda188b 0xda1dcc 0x6ec92f 0xa2aad8 0xa2bfcc 0xa2c370 0x7fbcf0 0x7fc19b 0x7fc285 0x7fc285 0x7fc285 0x7fc285 0x7fc4c1 0x9d6aca 0x9dd34f 0x9df975 0xac5b7d 0xac6e7f 0xa9198a
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
/usr/bin/mongod() [0xda1dcc]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5bf) [0x6ec92f]
/usr/bin/mongod(_ZNK5mongo5KeyV16toBsonEv+0x98) [0xa2aad8]
/usr/bin/mongod(_ZNK5mongo5KeyV113compareHybridERKS0_RKNS_8OrderingE+0x3c) [0xa2bfcc]
/usr/bin/mongod(_ZNK5mongo5KeyV19woCompareERKS0_RKNS_8OrderingE+0x230) [0xa2c370]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE4findERKNS_12IndexDetailsERKNS_5KeyV1ERKNS_7DiskLocERKNS_8OrderingERib+0x120) [0x7fbcf0]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x6b) [0x7fc19b]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13insertStepOneENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x155) [0x7fc285]
/usr/bin/mongod(_ZNK5mongo11BtreeBucketINS_12BtreeData_V1EE13twoStepInsertENS_7DiskLocERNS_30IndexInsertionContinuationImplIS1_EEb+0x1a1) [0x7fc4c1]
/usr/bin/mongod(_ZN5mongo18IndexInterfaceImplINS_12BtreeData_V1EE20beginInsertIntoIndexEiRNS_12IndexDetailsENS_7DiskLocERKNS_7BSONObjERKNS_8OrderingEb+0xda) [0x9d6aca]
/usr/bin/mongod(_ZN5mongo19fetchIndexInsertersERSt3setINS_7BSONObjENS_10BSONObjCmpESaIS1_EERNS_14IndexInterface13IndexInserterEPNS_16NamespaceDetailsEiRKS1_NS_7DiskLocEb+0x2cf) [0x9dd34f]
/usr/bin/mongod(_ZN5mongo24indexRecordUsingTwoStepsEPKcPNS_16NamespaceDetailsENS_7BSONObjENS_7DiskLocEb+0x175) [0x9df975]
/usr/bin/mongod(_ZN5mongo11DataFileMgr6insertEPKcPKvibbbPb+0x123d) [0xac5b7d]
/usr/bin/mongod(_ZN5mongo11DataFileMgr16insertWithObjModEPKcRNS_7BSONObjEbb+0x4f) [0xac6e7f]
/usr/bin/mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2eda) [0xa9198a]
Wed Mar 19 01:05:55.345 [repl writer worker 1] ERROR: writer worker caught exception: BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111 on: { ts: Timestamp 1395191155000|19, h: 8957154836310535201, v: 2, op: "i", ns:

<<<<Document Data Removed - size was about 3kb >>>>

Wed Mar 19 01:05:55.345 [repl writer worker 1] Fatal Assertion 16360
0xde0151 0xd9ff43 0xc28aac 0xdada91 0xe289d9 0x7f9c4d405e9a 0x7f9c4c7183fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xd9ff43]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28aac]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdada91]
/usr/bin/mongod() [0xe289d9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9c4d405e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9c4c7183fd]
Wed Mar 19 01:05:55.350 [repl writer worker 1]

***aborting after fassert() failure



 Comments   
Comment by Joel Moss [ 04/Aug/14 ]

This just happened to us too. We rean 2.4 in production for a good year until we upgraded to 2.6. We never had any issues with 2.4.

code
2014-08-04T09:46:05.194+0000 [repl writer worker 1] Assertion: 10334:BSONObj size: 16801042 (0x1005D12) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: ObjectId('53d93cc014db0332edf2aaec')
2014-08-04T09:46:05.200+0000 [repl writer worker 1] acv2.structures 0x11da801 0x117c9f9 0x11614b6 0x1161a0c 0x770e4b 0xc44c3b 0xc504b6 0xc3f807 0xe49393 0xeb21b9 0xeb2700 0x117029e 0x121f219 0x7f0c6e94d182 0x7f0c6dc5230d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x117c9f9]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11614b6]
/usr/bin/mongod() [0x1161a0c]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x41b) [0x770e4b]
/usr/bin/mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugEPNS_12UpdateDriverEPNS_14CanonicalQueryE+0x123b) [0xc44c3b]
/usr/bin/mongod(_ZN5mongo14UpdateExecutor7executeEv+0x66) [0xc504b6]
/usr/bin/mongod(_ZN5mongo6updateERKNS_13UpdateRequestEPNS_7OpDebugE+0x27) [0xc3f807]
/usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x17b3) [0xe49393]
/usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x749) [0xeb21b9]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb2700]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117029e]
/usr/bin/mongod() [0x121f219]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f0c6e94d182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0c6dc5230d]
2014-08-04T09:46:05.200+0000 [repl writer worker 1] ERROR: writer worker caught exception: :: caused by :: 10334 BSONObj size: 16801042 (0x1005D12) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: ObjectId('53d93cc014db0332edf2aaec') on: { ts: Timestamp 1407145155000|438, h: -6943971745641136233, v: 2, op: "u", ns: "acv2.structures", o2:

{ _id: ObjectId('53d93cc014db0332edf2aaec') }

, o: { $set: { children.v8.v8.third_party.icu.source.data.locales..svn.text-base.pa_Arab.txt.svn-base: { text-base: { ar_DJ.txt.svn-base: { prop-base: { mn.txt.svn-base: { pa_Arab.txt.svn-base: { ar_DJ.txt.svn-base: { text-base: { no_NO.txt.svn-base: { text-base: { sbp.txt.svn-base: { text-base: { nl_BE.txt.svn-base: { prop-base: { no_NO.txt.svn-base: { sbp.txt.svn-base: { nl_BE.txt.svn-base: { text-base: { or_IN.txt.svn-base: { text-base: { kw.txt.svn-base: { text-base: { shi_Tfng_MA.txt.svn-base: { prop-base: { or_IN.txt.svn-base: { kw.txt.svn-base: { shi_Tfng_MA.txt.svn-base: { text-base: { en_BB.txt.svn-base: { text-base: { vi_VN.txt.svn-base: { text-base: { om.txt.svn-base: { prop-base: { om.txt.svn-base: { text-base: { sh.txt.svn-base: { prop-base: { en_BB.txt.svn-base: { vi_VN.txt.svn-base: { text-base: { uz_AF.txt.svn-base: { prop-base: { sh.txt.svn-base: { text-base: { or.txt.svn-base: { text-base: { sr_Latn_RS.txt.svn-base: { prop-base: { uz_AF.txt.svn-base: { or.txt.svn-base: { text-base: { af_NA.txt.svn-base: { prop-base: { sr_Latn_RS.txt.svn-base: { text-base: { cy.txt.svn-base: { text-base: { ar_SO.txt.svn-base: { prop-base: { af_NA.txt.svn-base: { cy.txt.svn-base: { text-base: { en_MT.txt.svn-base: { prop-base: { ar_SO.txt.svn-base: { text-base: { ebu.txt.svn-base: { text-base: { ml_IN.txt.svn-base: { prop-base: { en_MT.txt.svn-base: { ebu.txt.svn-base: { text-base: { kkj_CM.txt.svn-base: { prop-base: { ml_IN.txt.svn-base: { text-base: { ko.txt.svn-base: { text-base: { en_GU.txt.svn-base: { prop-base: { kkj_CM.txt.svn-base: { ko.txt.svn-base: { text-base: { am.txt.svn-base: { text-base: { en_SS.txt.svn-base: { prop-base: { en_GU.txt.svn-base: { text-base: { ha_Latn_NG.txt.svn-base: { prop-base: { am.txt.svn-base: { en_SS.txt.svn-base: { text-base: { sr_XK.txt.svn-base: { text-base: { ka_GE.txt.svn-base:

{ ... }

} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }, $unset:

{ children.v8.v8.third_party.icu.source.data.locales..svn.tmp.text-base.pa_Arab.txt.svn-base: true }

} }
2014-08-04T09:46:05.200+0000 [repl writer worker 1] Fatal Assertion 16360
2014-08-04T09:46:05.200+0000 [repl writer worker 1]

***aborting after fassert() failure
code

Comment by Bruce Lucas (Inactive) [ 21/May/14 ]

Hi John,

As we haven't heard from you in some time, I'll go ahead and close this ticket out. Please feel free to re-open it if you would like to continue debugging this issue, possibly by private channel that I can provide as mentioned above.

Thanks,
Bruce

Comment by Bruce Lucas (Inactive) [ 09/May/14 ]

Hi John,

Just checking in to see whether you still want to pursue this issue, either in this ticket or in a private forum. Please let me know.

Thanks,
Bruce

Comment by Bruce Lucas (Inactive) [ 22/Apr/14 ]

Hi John,

It looks like there is probably some kind of corruption in your db; it is generally difficult to determine the cause of the corruption after the fact, but I would like to try. I understand that you may not still have this information, but if you could provide us with all available mongod logs relating to that node we may be able to spot something. Also, would you be willing to share the corrupted db in question with us for analysis? If you would like to share this information but wish to do so privately we can provide a private secure means for that.

Thanks,
Bruce

Comment by John Petrone [ 19/Mar/14 ]

Additional Info:

OS: Ubuntu 12.04.3 LTS

hosted on Amazon EC2

Generated at Thu Feb 08 03:31:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.