[SERVER-10703] Report corrupt namespace (collection or index) Created: 06/Sep/13 Updated: 06/Dec/22 Resolved: 15/Nov/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Logging, Storage |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Osmar Olivo | Assignee: | Backlog - Storage Execution Team |
| Resolution: | Done | Votes: | 3 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Storage Execution
|
| Participants: |
| Description |
|
When corruption is detected in the system. The logs report that there is corruption in the system, but they do not specify in which namespace the corruption was detected. I.E. if there is corruption in an index, this is reported, but the particular index or collection is not reported, thus you do not know which index to rebuild. |
| Comments |
| Comment by Eric Milkie [ 15/Nov/17 ] | |||||||||||||||||||||
|
The validate command in 3.4 now reports much more detailed information, including the index or collection name. | |||||||||||||||||||||
| Comment by Thiago Leite [ 10/Nov/16 ] | |||||||||||||||||||||
|
I had a similar issue(2 times) and could not solve without startup from a primary fresh copy . Nov 9 23:02:55 1m7 mongod.27018[62513]: [initandlisten] connection accepted from 10.235.204.59:60636 #27 (27 connections now open) , {"b":"400000","o":"ED61C8","s":"_ZN5mongo10logContextEPKc"}, {"b":"400000","o":"EC2028","s":"_ZN5mongo11msgassertedEiPKc"}, {"b":"400000","o":"EC20DC"}, {"b":"400000","o":"5C0229","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"}, {"b":"400000","o":"6CF2FF","s":"_ZN5mongo10Collection19aboutToDeleteCappedEPNS_16OperationContextERKNS_8RecordIdENS_10RecordDataE"}, {"b":"400000","o":"C737AC","s":"_ZN5mongo19CappedRecordStoreV111allocRecordEPNS_16OperationContextEib"}, {"b":"400000","o":"C6BDAE","s":"_ZN5mongo17RecordStoreV1Base13_insertRecordEPNS_16OperationContextEPKcib"}, {"b":"400000","o":"C6BFF9","s":"_ZN5mongo17RecordStoreV1Base12insertRecordEPNS_16OperationContextEPKcib"}, {"b":"400000","o":"BD99C5","s":"_ZN5mongo11RecordStore13insertRecordsEPNS_16OperationContextEPSt6vectorINS_6RecordESaIS4_EEb"}, {"b":"400000","o":"6D2B0E","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b"}, {"b":"400000","o":"6D3031","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb"}, {"b":"400000","o":"ABDBC4","s":"_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt6vectorINS_7BSONObjESaIS4_EE"}, {"b":"400000","o":"B565DB","s":"_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERKNS1_7OpQueueE"}, {"b":"400000","o":"B5859A","s":"_ZN5mongo4repl8SyncTail16oplogApplicationEv"}, {"b":"400000","o":"B4E250","s":"_ZN5mongo4repl13runSyncThreadEv"}, {"b":"400000","o":"175DC50","s":"execute_native_thread_routine"}, {"b":"3C58600000","o":"7851"}, {"b":"3C58200000","o":"E894D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-358.23.2.el6.x86_64", "version" : "#1 SMP Wed Oct 16 18:37:12 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4D5F33E10977D0D4B27FC98AA0326829E78A19A4" }, { "b" : "7FFF74AC5000", "elfType" : 3, "buildId" : "4D392D7A6140FA0AFF2F9098276ED6E94D137826" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B06F7B61A75BD941A6D9E36B2DC1CDCB4183D706" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0E46D8ED406D53C9A553C20859CD4679928AE7C0" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "756DBE5D1255F42B13E0659E3DD791D34A91465A" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "A38407EE35545AEA5CF08FE4CAA8B66E5909B6F3" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1686CCDAE5F8CED5A251E40074F55EFDF1688B75" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2E6E550A824EBC44AE5487B290A00923DB37761" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3FF31EFC5E0E5CFC4BFDAE19F3DE3AD55DA766CD" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "66B744D7D3B8201145C2C40E7A201F61B73E77D0" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DE7D207393D303AF233E6AD4D1E8A8314843422A" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "EFF68B7DE77D081BC4A0CB38FE9DCBC60541BF92" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "95EBB74C2C0A1E1714344036145A0239FFA4892D" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6ADE12F76961F73B33D160AC4D342222E7FC7A65" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D02E7D3149950118009A81997434E28B7D9EC9B2" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "5AFCBEA0D62EE0335714CCBAB7BA808E2A16028C" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6309B69A475D35D4E93D31DB3A8DDAF5100075C8" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" } ] }} mongod(ZN5mongo15printStackTraceERSo+0x32) [0x133aba2] mongod(_ZN5mongo10logContextEPKc+0x138) [0x12d61c8] mongod(_ZN5mongo11msgassertedEiPKc+0x88) [0x12c2028] mongod(+0xEC20DC) [0x12c20dc] mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x3B9) [0x9c0229] mongod(_ZN5mongo10Collection19aboutToDeleteCappedEPNS_16OperationContextERKNS_8RecordIdENS_10RecordDataE+0xBF) [0xacf2ff] mongod(_ZN5mongo19CappedRecordStoreV111allocRecordEPNS_16OperationContextEib+0x46C) [0x10737ac] mongod(_ZN5mongo17RecordStoreV1Base13_insertRecordEPNS_16OperationContextEPKcib+0x5E) [0x106bdae] mongod(_ZN5mongo17RecordStoreV1Base12insertRecordEPNS_16OperationContextEPKcib+0xA9) [0x106bff9] mongod(_ZN5mongo11RecordStore13insertRecordsEPNS_16OperationContextEPSt6vectorINS_6RecordESaIS4_EEb+0xB5) [0xfd99c5] mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9gnu_cxx17normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x16E) [0xad2b0e] mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9gnu_cxx17_normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xad3031] mongod(_ZN5mongo4repl15writeOpsToOplogEPNS_16OperationContextERKSt6vectorINS_7BSONObjESaIS4_EE+0x144) [0xebdbc4] mongod(_ZN5mongo4repl8SyncTail10multiApplyEPNS_16OperationContextERKNS1_7OpQueueE+0x86B) [0xf565db] mongod(_ZN5mongo4repl8SyncTail16oplogApplicationEv+0x136A) [0xf5859a] mongod(_ZN5mongo4repl13runSyncThreadEv+0x2B0) [0xf4e250] mongod(execute_native_thread_routine+0x20) [0x1b5dc50] libpthread.so.0(+0x7851) [0x3c58607851] libc.so.6(clone+0x6D) [0x3c582e894d] ----- END BACKTRACE ----- , {"b":"400000","o":"F3A4D2"}, {"b":"400000","o":"17159F6","s":"_ZN10__cxxabiv111__terminateEPFvvE"}, {"b":"400000","o":"1715A23"}, {"b":"400000","o":"B4E388","s":"_ZN5mongo4repl13runSyncThreadEv"}, {"b":"400000","o":"175DC50","s":"execute_native_thread_routine"}, {"b":"3C58600000","o":"7851"}, {"b":"3C58200000","o":"E894D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-358.23.2.el6.x86_64", "version" : "#1 SMP Wed Oct 16 18:37:12 UTC 2013", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "4D5F33E10977D0D4B27FC98AA0326829E78A19A4" }, { "b" : "7FFF74AC5000", "elfType" : 3, "buildId" : "4D392D7A6140FA0AFF2F9098276ED6E94D137826" }, { "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "B06F7B61A75BD941A6D9E36B2DC1CDCB4183D706" }, { "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "0E46D8ED406D53C9A553C20859CD4679928AE7C0" }, { "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "756DBE5D1255F42B13E0659E3DD791D34A91465A" }, { "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "A38407EE35545AEA5CF08FE4CAA8B66E5909B6F3" }, { "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "1686CCDAE5F8CED5A251E40074F55EFDF1688B75" }, { "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "A2E6E550A824EBC44AE5487B290A00923DB37761" }, { "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3FF31EFC5E0E5CFC4BFDAE19F3DE3AD55DA766CD" }, { "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "66B744D7D3B8201145C2C40E7A201F61B73E77D0" }, { "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "DE7D207393D303AF233E6AD4D1E8A8314843422A" }, { "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "EFF68B7DE77D081BC4A0CB38FE9DCBC60541BF92" }, { "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "95EBB74C2C0A1E1714344036145A0239FFA4892D" }, { "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "6ADE12F76961F73B33D160AC4D342222E7FC7A65" }, { "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "D02E7D3149950118009A81997434E28B7D9EC9B2" }, { "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "5AFCBEA0D62EE0335714CCBAB7BA808E2A16028C" }, { "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8A8734DC37305D8CC2EF8F8C3E5EA03171DB07EC" }, { "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6309B69A475D35D4E93D31DB3A8DDAF5100075C8" }, { "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "BAD5C71361DADF259B6E306A49E6F47F24AEA3DC" }] }} mongod(ZN5mongo15printStackTraceERSo+0x32) [0x133aba2] mongod(+0xF3A4D2) [0x133a4d2] mongod(_ZN10cxxabiv111_terminateEPFvvE+0x6) [0x1b159f6] mongod(+0x1715A23) [0x1b15a23] mongod(_ZN5mongo4repl13runSyncThreadEv+0x3E8) [0xf4e388] mongod(execute_native_thread_routine+0x20) [0x1b5dc50] libpthread.so.0(+0x7851) [0x3c58607851] libc.so.6(clone+0x6D) [0x3c582e894d] ----- END BACKTRACE ----- Version: Note: When I started the node without replicaset parameter the mongod was started with sucess, but After I put the replicaset parameter back the same error started to ocorr again. | |||||||||||||||||||||
| Comment by Alexander Komyagin [ 22/Nov/13 ] | |||||||||||||||||||||
|
Here is an example. According to the stack trace, it failed on the unindexRecord op. So it's not the oplog problem, I think. It would be helpful to know which collection/what index was accessed.
| |||||||||||||||||||||
| Comment by Eliot Horowitz (Inactive) [ 07/Sep/13 ] | |||||||||||||||||||||
|
Please be explicit with an example. |