-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
None
-
Affects Version/s: 2.6.12
-
Component/s: Storage
-
None
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
I am running mongodb 2.6.12 on ubuntu 64bits. We have an instance with more than 10TB of scientific research data, a python app is using the db via pymongo. Suddenly, after a db compact the database I was/am unable to run a mongodump because of errors like this:
2017-01-19T15:30:34.194-0800 [conn1] Assertion: 10334:BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108
2017-01-19T15:30:34.208-0800 [conn1] foobar.calls 0x121df81 0x11bd689 0x11a23e6 0x11a294c 0x7762d3 0xf1a158 0xaa9fd0 0xab90ab 0xd902ef 0xd73b87 0xbb4562 0xbb5ae0 0x774658 0x11d18db 0x7f6b0e61c184 0x7f6b0d92137d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x121df81]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x11bd689]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11a23e6]
/usr/bin/mongod() [0x11a294c]
/usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x473) [0x7762d3]
/usr/bin/mongod(_ZNK5mongo7DiskLoc3objEv+0x68) [0xf1a158]
/usr/bin/mongod(_ZN5mongo14CollectionScan4workEPm+0x2b0) [0xaa9fd0]
/usr/bin/mongod(_ZN5mongo10LimitStage4workEPm+0x5b) [0xab90ab]
/usr/bin/mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_7DiskLocE+0xef) [0xd902ef]
/usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x977) [0xd73b87]
/usr/bin/mongod() [0xbb4562]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x500) [0xbb5ae0]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x98) [0x774658]
/usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x50b) [0x11d18db]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f6b0e61c184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6b0d92137d]
2017-01-19T15:30:34.208-0800 [conn1] assertion 10334 BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108 ns:foobar.calls query:{ arguments.value: /type=108/ }
2017-01-19T15:30:34.208-0800 [conn1] ntoskip:0 ntoreturn:-1
2017-01-19T15:30:34.208-0800 [conn1] query foobar.calls query: { arguments.value: /type=108/ } keyUpdates:0 exception: BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ���̀��Y�A���dc��(�����Z ��f��(�A�@AcU��:��|��T�}
���'c�: ?type=108 code:10334 numYields:0 locks(micros) r:18870 reslen:237 18ms}}
I also tried to run a db repair, but I got this:
db.repairDatabase({backupOriginalFiles: true})
{
"errmsg" : "exception: BSONObj size: 1732548332 (0x674496EC) is invalid. Size must be between 0 and 16793600(16MB) First element: &ۺ����\u0006�̀��Y�A��\u0004�dc��(��������Z\t��f��(��A�@AcU\u0004�\u0010�:��|\u000f��T��}\n���'�\bc�: ?type=108",
"code" : 10334,
"ok" : 0
}
So now we are unable to fix the db, someone can help? Unfortunately, we don't have many foundings, so we were unable to pay a backup server for so much data.