[SERVER-10703] Report corrupt namespace (collection or index) Created: 06/Sep/13  Updated: 06/Dec/22  Resolved: 15/Nov/17

Status: Closed
Project: Core Server
Component/s: Logging, Storage
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Osmar Olivo Assignee: Backlog - Storage Execution Team
Resolution: Done Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Storage Execution
Participants:

 Description   

When corruption is detected in the system. The logs report that there is corruption in the system, but they do not specify in which namespace the corruption was detected.

I.E. if there is corruption in an index, this is reported, but the particular index or collection is not reported, thus you do not know which index to rebuild.



 Comments   
Comment by Eric Milkie [ 15/Nov/17 ]

The validate command in 3.4 now reports much more detailed information, including the index or collection name.

Comment by Thiago Leite [ 10/Nov/16 ]

I had a similar issue(2 times) and could not solve without startup from a primary fresh copy .

Nov 9 23:02:55 1m7 mongod.27018[62513]: [initandlisten] connection accepted from 10.235.204.59:60636 #27 (27 connections now open)
Nov 9 23:02:55 1m7 mongod.27018[62513]: [conn27] Successfully authenticated as principal __system on local
Nov 9 23:02:55 1m7 mongod.27018[62513]: [initandlisten] connection accepted from 10.235.137.24:52884 #28 (28 connections now open)
Nov 9 23:02:55 1m7 mongod.27018[62513]: [conn28] Successfully authenticated as principal __system on local
Nov 9 23:02:56 1m7 mongod.27018[62513]: [initandlisten] connection accepted from 10.253.124.86:34983 #29 (29 connections now open)
Nov 9 23:02:56 1m7 mongod.27018[62513]: [conn29] Successfully authenticated as principal __system on local
Nov 9 23:02:56 1m7 mongod.27018[62513]: [rsSync] Assertion: 10334:BSONObj size: 1952539658 (0x7461640A) is invalid. Size must be between 0 and 16793600(16MB) First element: _inicio: ?type=97
Nov 9 23:02:56 1m7 mongod.27018[62513]: [rsSync] 0x133aba2 0x12d61c8 0x12c2028 0x12c20dc 0x9c0229 0xacf2ff 0x10737ac 0x106bdae 0x106bff9 0xfd99c5 0xad2b0e 0xad3031 0xebdbc4 0xf565db 0xf5859a 0xf4e250 0x1b5dc50 0x3c58607851 0x3c582e894d ----- BEGIN BACKTRACE ----- {"backtrace":[

{"b":"400000","o":"F3ABA2","s":"_ZN5mongo15printStackTraceERSo"}

,

{"b":"400000","o":"ED61C8","s":"_ZN5mongo10logContextEPKc"}

,

{"b":"400000","o":"EC2028","s":"_ZN5mongo11msgassertedEiPKc"}

,

{"b":"400000","o":"EC20DC"}

,

{"b":"400000","o":"5C0229","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"}

,

{"b":"400000","o":"6CF2FF","s":"_ZN5mongo10Collection19aboutToDeleteCappedEPNS_16OperationContextERKNS_8RecordIdENS_10RecordDataE"}

,

{"b":"400000","o":"C737AC","s":"_ZN5mongo19CappedRecordStoreV111allocRecordEPNS_16OperationContextEib"}

,

{"b":"400000","o":"C6BDAE","s":"_ZN5mongo17RecordStoreV1Base13_insertRecordEPNS_16OperationContextEPKcib"}

,

{"b":"400000","o":"C6BFF9","s":"_ZN5mongo17RecordStoreV1Base12insertRecordEPNS_16OperationContextEPKcib"}

,

{"b":"400000","o":"BD99C5","s":"_ZN5mongo11RecordStore13insertRecordsEPNS_16OperationContextEPSt6vectorINS_6RecordESaIS4_EEb"}

,

{"b":"400000","o":"6D2B0E","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b"}

,

{"b":"400000","o":"6D3031","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb"}

,

{"b":"400000","o":"ABDBC4","s":"_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt6vectorINS_7BSONObjESaIS4_EE"}

,

{"b":"400000","o":"B565DB","s":"_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERKNS1_7OpQueueE"}

,

{"b":"400000","o":"B5859A","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEv"}

,

{"b":"400000","o":"B4E250","s":"_ZN5mongo4repl13runSyncThreadEv"}

,

{"b":"400000","o":"175DC50","s":"execute_native_thread_routine"}

,

{"b":"3C58600000","o":"7851"}

,

{"b":"3C58200000","o":"E894D","s":"clone"}

],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "2.6.32-358.23.2.el6.x86_64", "version" : "#1 SMP Wed Oct 16 18:37:12 UTC 2013", "machine" : "x86_64" }

, "somap" : [

{ "elfType" : 2, "b" : "400000", "buildId" : "4D5F33E10977D0D4B27FC98AA0326829E78A19A4" }

,

{ "b" : "7FFF74AC5000", "elfType" : 3, "buildId" : "4D392D7A6140FA0AFF2F9098276ED6E94D137826" }

,

{ "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B06F7B61A75BD941A6D9E36B2DC1CDCB4183D706" }

,

{ "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0E46D8ED406D53C9A553C20859CD4679928AE7C0" }

,

{ "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "756DBE5D1255F42B13E0659E3DD791D34A91465A" }

,

{ "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "A38407EE35545AEA5CF08FE4CAA8B66E5909B6F3" }

,

{ "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1686CCDAE5F8CED5A251E40074F55EFDF1688B75" }

,

{ "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2E6E550A824EBC44AE5487B290A00923DB37761" }

,

{ "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3FF31EFC5E0E5CFC4BFDAE19F3DE3AD55DA766CD" }

,

{ "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "66B744D7D3B8201145C2C40E7A201F61B73E77D0" }

,

{ "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DE7D207393D303AF233E6AD4D1E8A8314843422A" }

,

{ "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "EFF68B7DE77D081BC4A0CB38FE9DCBC60541BF92" }

,

{ "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "95EBB74C2C0A1E1714344036145A0239FFA4892D" }

,

{ "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6ADE12F76961F73B33D160AC4D342222E7FC7A65" }

,

{ "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D02E7D3149950118009A81997434E28B7D9EC9B2" }

,

{ "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }

,

{ "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "5AFCBEA0D62EE0335714CCBAB7BA808E2A16028C" }

,

{ "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }

,

{ "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6309B69A475D35D4E93D31DB3A8DDAF5100075C8" }

,

{ "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" }

] }} mongod(ZN5mongo15printStackTraceERSo+0x32) [0x133aba2] mongod(_ZN5mongo10logContextEPKc+0x138) [0x12d61c8] mongod(_ZN5mongo11msgassertedEiPKc+0x88) [0x12c2028] mongod(+0xEC20DC) [0x12c20dc] mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3B9) [0x9c0229] mongod(_ZN5mongo10Collection19aboutToDeleteCappedEPNS_16OperationContextERKNS_8RecordIdENS_10RecordDataE+0xBF) [0xacf2ff] mongod(_ZN5mongo19CappedRecordStoreV111allocRecordEPNS_16OperationContextEib+0x46C) [0x10737ac] mongod(_ZN5mongo17RecordStoreV1Base13_insertRecordEPNS_16OperationContextEPKcib+0x5E) [0x106bdae] mongod(_ZN5mongo17RecordStoreV1Base12insertRecordEPNS_16OperationContextEPKcib+0xA9) [0x106bff9] mongod(_ZN5mongo11RecordStore13insertRecordsEPNS_16OperationContextEPSt6vectorINS_6RecordESaIS4_EEb+0xB5) [0xfd99c5] mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9gnu_cxx17normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x16E) [0xad2b0e] mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9gnu_cxx17_normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xad3031] mongod(_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt6vectorINS_7BSONObjESaIS4_EE+0x144) [0xebdbc4] mongod(_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERKNS1_7OpQueueE+0x86B) [0xf565db] mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEv+0x136A) [0xf5859a] mongod(_ZN5mongo4repl13runSyncThreadEv+0x2B0) [0xf4e250] mongod(execute_native_thread_routine+0x20) [0x1b5dc50] libpthread.so.0(+0x7851) [0x3c58607851] libc.so.6(clone+0x6D) [0x3c582e894d] ----- END BACKTRACE -----
Nov 9 23:02:56 1m7 mongod.27018[62513]: [rsSync] terminate() called. An exception is active; attempting to gather more information
Nov 9 23:02:56 1m7 mongod.27018[62513]: [rsSync] DBException::toString(): 10334 BSONObj size: 1952539658 (0x7461640A) is invalid. Size must be between 0 and 16793600(16MB) First element: _inicio: ?type=97 Actual exception type: mongo::MsgAssertionException 0x133aba2 0x133a4d2 0x1b159f6 0x1b15a23 0xf4e388 0x1b5dc50 0x3c58607851 0x3c582e894d ----- BEGIN BACKTRACE ----- {"backtrace":[

{"b":"400000","o":"F3ABA2","s":"_ZN5mongo15printStackTraceERSo"}

,

{"b":"400000","o":"F3A4D2"}

,

{"b":"400000","o":"17159F6","s":"_ZN10__cxxabiv111__terminateEPFvvE"}

,

{"b":"400000","o":"1715A23"}

,

{"b":"400000","o":"B4E388","s":"_ZN5mongo4repl13runSyncThreadEv"}

,

{"b":"400000","o":"175DC50","s":"execute_native_thread_routine"}

,

{"b":"3C58600000","o":"7851"}

,

{"b":"3C58200000","o":"E894D","s":"clone"}

],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" :

{ "sysname" : "Linux", "release" : "2.6.32-358.23.2.el6.x86_64", "version" : "#1 SMP Wed Oct 16 18:37:12 UTC 2013", "machine" : "x86_64" }

, "somap" : [

{ "elfType" : 2, "b" : "400000", "buildId" : "4D5F33E10977D0D4B27FC98AA0326829E78A19A4" }

,

{ "b" : "7FFF74AC5000", "elfType" : 3, "buildId" : "4D392D7A6140FA0AFF2F9098276ED6E94D137826" }

,

{ "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B06F7B61A75BD941A6D9E36B2DC1CDCB4183D706" }

,

{ "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0E46D8ED406D53C9A553C20859CD4679928AE7C0" }

,

{ "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "756DBE5D1255F42B13E0659E3DD791D34A91465A" }

,

{ "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "A38407EE35545AEA5CF08FE4CAA8B66E5909B6F3" }

,

{ "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1686CCDAE5F8CED5A251E40074F55EFDF1688B75" }

,

{ "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2E6E550A824EBC44AE5487B290A00923DB37761" }

,

{ "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3FF31EFC5E0E5CFC4BFDAE19F3DE3AD55DA766CD" }

,

{ "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "66B744D7D3B8201145C2C40E7A201F61B73E77D0" }

,

{ "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DE7D207393D303AF233E6AD4D1E8A8314843422A" }

,

{ "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "EFF68B7DE77D081BC4A0CB38FE9DCBC60541BF92" }

,

{ "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "95EBB74C2C0A1E1714344036145A0239FFA4892D" }

,

{ "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6ADE12F76961F73B33D160AC4D342222E7FC7A65" }

,

{ "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D02E7D3149950118009A81997434E28B7D9EC9B2" }

,

{ "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }

,

{ "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "5AFCBEA0D62EE0335714CCBAB7BA808E2A16028C" }

,

{ "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }

,

{ "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6309B69A475D35D4E93D31DB3A8DDAF5100075C8" }

,

{ "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" }

] }} mongod(ZN5mongo15printStackTraceERSo+0x32) [0x133aba2] mongod(+0xF3A4D2) [0x133a4d2] mongod(_ZN10cxxabiv111_terminateEPFvvE+0x6) [0x1b159f6] mongod(+0x1715A23) [0x1b15a23] mongod(_ZN5mongo4repl13runSyncThreadEv+0x3E8) [0xf4e388] mongod(execute_native_thread_routine+0x20) [0x1b5dc50] libpthread.so.0(+0x7851) [0x3c58607851] libc.so.6(clone+0x6D) [0x3c582e894d] ----- END BACKTRACE -----

Version:
MongoDB shell version: 3.2.8

Note: When I started the node without replicaset parameter the mongod was started with sucess, but After I put the replicaset parameter back the same error started to ocorr again.

Comment by Alexander Komyagin [ 22/Nov/13 ]

Here is an example. According to the stack trace, it failed on the unindexRecord op. So it's not the oplog problem, I think. It would be helpful to know which collection/what index was accessed.

Sat Nov  9 23:35:08.561 [rsSync] Assertion: 10334:BSONObj size: -394360188 (0x848A7EE8) is invalid. Size must be between 0 and 16793600(16MB) First element: blablabla : ?type=-16
0xdcf361 0xd90a1b 0xd90f5c 0x6eaf99 0x9d9e50 0xac177f 0x815826 0xa59cd9 0xa59d1c 0xab3b5c 0xa6d73c 0xc198db 0xc212b4 0xc21858 0xc218ad 0xc21b7c 0xe17cb9 0x38160077f1 0x3815ce5ccd 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
 /usr/bin/mongod() [0xd90f5c]
 /usr/bin/mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x5a9) [0x6eaf99]
 /usr/bin/mongod(_ZN5mongo13unindexRecordEPNS_16NamespaceDetailsEPNS_6RecordERKNS_7DiskLocEb+0x130) [0x9d9e50]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr12deleteRecordEPNS_16NamespaceDetailsEPKcPNS_6RecordERKNS_7DiskLocEbbb+0x1bf) [0xac177f]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails11cappedAllocEPKci+0x436) [0x815826]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails6_allocEPKci+0x29) [0xa59cd9]
 /usr/bin/mongod(_ZN5mongo16NamespaceDetails5allocEPKci+0x3c) [0xa59d1c]
 /usr/bin/mongod(_ZN5mongo11DataFileMgr17fast_oplog_insertEPNS_16NamespaceDetailsEPKci+0x1ec) [0xab3b5c]
 /usr/bin/mongod(_ZN5mongo11_logOpObjRSERKNS_7BSONObjE+0x4cc) [0xa6d73c]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail15applyOpsToOplogEPSt5dequeINS_7BSONObjESaIS3_EE+0x4b) [0xc198db]
 /usr/bin/mongod(_ZN5mongo7replset8SyncTail16oplogApplicationEv+0x584) [0xc212b4]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl11_syncThreadEv+0xb8) [0xc21858]
 /usr/bin/mongod(_ZN5mongo11ReplSetImpl10syncThreadEv+0x2d) [0xc218ad]
 /usr/bin/mongod(_ZN5mongo15startSyncThreadEv+0x6c) [0xc21b7c]
 /usr/bin/mongod() [0xe17cb9]
 /lib64/libpthread.so.0() [0x38160077f1]
 /lib64/libc.so.6(clone+0x6d) [0x3815ce5ccd]

Comment by Eliot Horowitz (Inactive) [ 07/Sep/13 ]

Please be explicit with an example.

Generated at Thu Feb 08 03:23:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.