[SERVER-13279] Replica Set Secondary crashes with invalid size error Size must be between 0 and 16793600(16MB) Created: 19/Mar/14 Updated: 07/Aug/14 Resolved: 21/May/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 2.4.7 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | John Petrone | Assignee: | Bruce Lucas (Inactive) |
| Resolution: | Cannot Reproduce | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | Unclear, although once it occurred could not bring mongod back up as it failed the same assertion on the same document every time. |
| Participants: |
| Description |
|
Three member replica set, running version 2.4.7. Secondary died with an assert failure on invalid document size (Size must be between 0 and 16793600(16MB)). Two attempts to restart the secondary failed with the same error, on the same document. No issues at all with other secondary or primary. Ended up creating new storage volume and performing a resync to bring it back online. I've included the info from the mongodb.log below. The document that had the invalid document size had just been inserted into the primary and replicated to the secondaries. Document looks fine on the primary and the surviving secondary. Actual document size is about 3kb. Log info: Wed Mar 19 01:05:55.184 [repl prefetch worker] Assertion: 10334:BSONObj size: 1811939328 (0x0000006C) is invalid. Size must be between 0 and 16793600(16MB) First element: Bo²: ?type=111 <<<<Document Data Removed - size was about 3kb >>>> Wed Mar 19 01:05:55.345 [repl writer worker 1] Fatal Assertion 16360 ***aborting after fassert() failure |
| Comments |
| Comment by Joel Moss [ 04/Aug/14 ] |
|
This just happened to us too. We rean 2.4 in production for a good year until we upgraded to 2.6. We never had any issues with 2.4. code , o: { $set: { children.v8.v8.third_party.icu.source.data.locales..svn.text-base.pa_Arab.txt.svn-base: { text-base: { ar_DJ.txt.svn-base: { prop-base: { mn.txt.svn-base: { pa_Arab.txt.svn-base: { ar_DJ.txt.svn-base: { text-base: { no_NO.txt.svn-base: { text-base: { sbp.txt.svn-base: { text-base: { nl_BE.txt.svn-base: { prop-base: { no_NO.txt.svn-base: { sbp.txt.svn-base: { nl_BE.txt.svn-base: { text-base: { or_IN.txt.svn-base: { text-base: { kw.txt.svn-base: { text-base: { shi_Tfng_MA.txt.svn-base: { prop-base: { or_IN.txt.svn-base: { kw.txt.svn-base: { shi_Tfng_MA.txt.svn-base: { text-base: { en_BB.txt.svn-base: { text-base: { vi_VN.txt.svn-base: { text-base: { om.txt.svn-base: { prop-base: { om.txt.svn-base: { text-base: { sh.txt.svn-base: { prop-base: { en_BB.txt.svn-base: { vi_VN.txt.svn-base: { text-base: { uz_AF.txt.svn-base: { prop-base: { sh.txt.svn-base: { text-base: { or.txt.svn-base: { text-base: { sr_Latn_RS.txt.svn-base: { prop-base: { uz_AF.txt.svn-base: { or.txt.svn-base: { text-base: { af_NA.txt.svn-base: { prop-base: { sr_Latn_RS.txt.svn-base: { text-base: { cy.txt.svn-base: { text-base: { ar_SO.txt.svn-base: { prop-base: { af_NA.txt.svn-base: { cy.txt.svn-base: { text-base: { en_MT.txt.svn-base: { prop-base: { ar_SO.txt.svn-base: { text-base: { ebu.txt.svn-base: { text-base: { ml_IN.txt.svn-base: { prop-base: { en_MT.txt.svn-base: { ebu.txt.svn-base: { text-base: { kkj_CM.txt.svn-base: { prop-base: { ml_IN.txt.svn-base: { text-base: { ko.txt.svn-base: { text-base: { en_GU.txt.svn-base: { prop-base: { kkj_CM.txt.svn-base: { ko.txt.svn-base: { text-base: { am.txt.svn-base: { text-base: { en_SS.txt.svn-base: { prop-base: { en_GU.txt.svn-base: { text-base: { ha_Latn_NG.txt.svn-base: { prop-base: { am.txt.svn-base: { en_SS.txt.svn-base: { text-base: { sr_XK.txt.svn-base: { text-base: { ka_GE.txt.svn-base: { ... }} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }, $unset: { children.v8.v8.third_party.icu.source.data.locales..svn.tmp.text-base.pa_Arab.txt.svn-base: true } } } ***aborting after fassert() failure |
| Comment by Bruce Lucas (Inactive) [ 21/May/14 ] |
|
Hi John, As we haven't heard from you in some time, I'll go ahead and close this ticket out. Please feel free to re-open it if you would like to continue debugging this issue, possibly by private channel that I can provide as mentioned above. Thanks, |
| Comment by Bruce Lucas (Inactive) [ 09/May/14 ] |
|
Hi John, Just checking in to see whether you still want to pursue this issue, either in this ticket or in a private forum. Please let me know. Thanks, |
| Comment by Bruce Lucas (Inactive) [ 22/Apr/14 ] |
|
Hi John, It looks like there is probably some kind of corruption in your db; it is generally difficult to determine the cause of the corruption after the fact, but I would like to try. I understand that you may not still have this information, but if you could provide us with all available mongod logs relating to that node we may be able to spot something. Also, would you be willing to share the corrupted db in question with us for analysis? If you would like to share this information but wish to do so privately we can provide a private secure means for that. Thanks, |
| Comment by John Petrone [ 19/Mar/14 ] |
|
Additional Info: OS: Ubuntu 12.04.3 LTS hosted on Amazon EC2 |