[SERVER-27765] MongoDB, python and pymongo: BSONObj size is invalid Created: 19/Jan/17  Updated: 24/Jan/17  Resolved: 21/Jan/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: 2.6.12
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Antani Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

I am running mongodb 2.6.12 on ubuntu 64bits. We have an instance with more than 10TB of scientific research data, a python app is using the db via pymongo. Suddenly, after a db compact the database I was/am unable to run a mongodump because of errors like this:

2017-01-19T15:30:34.194-0800 [conn1] Assertion: 10334:BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z       ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108
2017-01-19T15:30:34.208-0800 [conn1] foobar.calls 0x121df81 0x11bd689 0x11a23e6 0x11a294c 0x7762d3 0xf1a158 0xaa9fd0 0xab90ab 0xd902ef 0xd73b87 0xbb4562 0xbb5ae0 0x774658 0x11d18db 0x7f6b0e61c184 0x7f6b0d92137d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x121df81]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x11bd689]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11a23e6]
 /usr/bin/mongod() [0x11a294c]
 /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x473) [0x7762d3]
 /usr/bin/mongod(_ZNK5mongo7DiskLoc3objEv+0x68) [0xf1a158]
 /usr/bin/mongod(_ZN5mongo14CollectionScan4workEPm+0x2b0) [0xaa9fd0]
 /usr/bin/mongod(_ZN5mongo10LimitStage4workEPm+0x5b) [0xab90ab]
 /usr/bin/mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_7DiskLocE+0xef) [0xd902ef]
 /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x977) [0xd73b87]
 /usr/bin/mongod() [0xbb4562]
 /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x500) [0xbb5ae0]
 /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x774658]
 /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x50b) [0x11d18db]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6b0e61c184]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6b0d92137d]
2017-01-19T15:30:34.208-0800 [conn1] assertion 10334 BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z        ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108 ns:foobar.calls query:{ arguments.value: /type=108/ }
2017-01-19T15:30:34.208-0800 [conn1]  ntoskip:0 ntoreturn:-1
2017-01-19T15:30:34.208-0800 [conn1] query foobar.calls query: { arguments.value: /type=108/ } keyUpdates:0 exception: BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108 code:10334 numYields:0 locks(micros) r:18870 reslen:237 18ms}}

I also tried to run a db repair, but I got this:

db.repairDatabase({backupOriginalFiles: true})
{
        "errmsg" : "exception: BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ����\u0006�̀��Y�A��\u0004�dc��(��������Z\t��f��(��A�@AcU\u0004�\u0010�:��|\u000f��T��}\n���'�\bc�: ?type=108",
        "code" : 10334,
        "ok" : 0
}

So now we are unable to fix the db, someone can help? Unfortunately, we don't have many foundings, so we were unable to pay a backup server for so much data.



 Comments   
Comment by Kelsey Schubert [ 24/Jan/17 ]

Hi antan23,

Please continue to run the mongodump operation until it completes. In some situations, the estimate provided by the progress bar may not be accurate.

Thank you,
Thomas

Comment by Antani [ 21/Jan/17 ]

Thanks for your reply.

I already stopped the mongo deamon and tried a:

mongodump --repair --dbpath /path/to/db --db dbname --forceTableScan

But after reaching the 100% it was going on and I stopped it at 114% (at that point I thought it was not working.
Yep we want to upgrade to 3.x but before we have to recover our data.

I am not sure if mongod should be stopped running this operation or why this is going over 100%, any hint?

Thanks so much.

Comment by Kelsey Schubert [ 20/Jan/17 ]

Hi antan23,

Unfortunately, this error indicates that the data became corrupted in some way. It's not clear whether this due to a faulty storage layer or some other issue. In cases like this identifying the root cause can be very challenging without a reproduction. Please note that MongoDB 2.6.12 is no longer supported, and we recommend upgrading to a newer version.

I see that the repairDatabase command was unsuccessful at recovering your data. Therefore, my advice would be to create a copy of your current $dbpath, and execute the following command as a last effort.

mongodump --repair --dbpath /path/to/db --db dbname

mongodump in MongoDB 2.6 utilizes a different algorithm to repair corruption, which may be able to recover some of your data.

For more details about executing this command, please review our documentation. Please be aware that some manual intervention may be required to deduplicate documents after this process completes and there is no guarantee that this operation will recover all or most of your data depending on the extent of corruption.

I understand that there are costs associated with the backup methods we recommend. By upgrading to MongoDB 3.0 or later, you would be able to take advantage of WiredTiger's Compression to reduce the data size of your files on disk, and, consequently, the associated costs to store backups.

Kind regards,
Thomas

Generated at Thu Feb 08 04:16:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.