[SERVER-28226] WT Crash Created: 07/Mar/17  Updated: 15/Nov/21  Resolved: 21/Mar/17

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: hu dai Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: envm, rns, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS release 6.6 (Final)


Operating System: Linux
Participants:

 Description   

2017-03-06T20:02:22.505+0800 I NETWORK  [conn4063980] end connection 192.168.xx.33:44020 (102 connections now open)
2017-03-06T20:02:22.891+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1488801742:891740][28613:0x7f37b5105700], file:apollo_modb/index-84-9204097186848499981.wt, WT_CURSOR.remove: read checksum error for 4096B block at offset 95326208: calculated block checksum of 1332835297 doesn't match expected checksum of 3877724731
2017-03-06T20:02:22.891+0800 E STORAGE  [repl writer worker 1] WiredTiger (0) [1488801742:891802][28613:0x7f37b5105700], file:apollo_modb/index-84-9204097186848499981.wt, WT_CURSOR.remove: apollo_modb/index-84-9204097186848499981.wt: encountered an illegal file format or internal value
2017-03-06T20:02:22.891+0800 E STORAGE  [repl writer worker 1] WiredTiger (-31804) [1488801742:891813][28613:0x7f37b5105700], file:apollo_modb/index-84-9204097186848499981.wt, WT_CURSOR.remove: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-03-06T20:02:22.891+0800 I -        [repl writer worker 1] Fatal Assertion 28558
2017-03-06T20:02:22.891+0800 I -        [repl writer worker 1]
 
***aborting after fassert() failure
 
 
2017-03-06T20:02:22.891+0800 I -        [rsSync] Fatal Assertion 28559
2017-03-06T20:02:22.891+0800 I -        [rsSync]
 
***aborting after fassert() failure
 
 
2017-03-06T20:02:22.970+0800 I -        [WTJournalFlusher] Fatal Assertion 28559
2017-03-06T20:02:22.970+0800 I -        [WTJournalFlusher]
 
***aborting after fassert() failure
 
 
2017-03-06T20:02:23.000+0800 I -        [ftdc] Fatal Assertion 28559
2017-03-06T20:02:23.000+0800 I -        [ftdc]
 
***aborting after fassert() failure
 
 
2017-03-06T20:02:23.027+0800 F -        [repl writer worker 1] Got signal: 6 (Aborted).
 
 0x13403e2 0x133f319 0x133fb22 0x7f37cff27710 0x7f37cfbb6625 0x7f37cfbb7e05 0x12c3602 0x10b3c63 0x1aad81c 0x1aadc9d 0x1aae084 0x19ddff2 0x19fb50d 0x1a00ab2 0x1a20b27 0x19f2244 0x1a3cc0a 0x1090a85 0x108f2f7 0xcade28 0xad7384 0xc23611 0xc23ff8 0xe32a05 0xe330c9 0xe331c5 0xd3d569 0xec4374 0xf5ac0e 0xf54a10 0xf56656 0xf57256 0xf5a55b 0x12ce131 0x12cea99 0x12cf5f0 0x1b69250 0x7f37cff1f9d1 0x7f37cfc6c8fd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F403E2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F3F319"},{"b":"400000","o":"F3FB22"},{"b":"7F37CFF18000","o":"F710"},{"b":"7F37CFB84000","o":"32625","s":"gsignal"},{"b":"7F37CFB84000","o":"33E05","s":"abort"},{"b":"400000","o":"EC3602","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CB3C63"},{"b":"400000","o":"16AD81C","s":"__wt_eventv"},{"b":"400000","o":"16ADC9D","s":"__wt_err"},{"b":"400000","o":"16AE084","s":"__wt_panic"},{"b":"400000","o":"15DDFF2","s":"__wt_bm_read"},{"b":"400000","o":"15FB50D","s":"__wt_bt_read"},{"b":"400000","o":"1600AB2","s":"__wt_page_in_func"},{"b":"400000","o":"1620B27","s":"__wt_row_search"},{"b":"400000","o":"15F2244","s":"__wt_btcur_remove"},{"b":"400000","o":"163CC0A"},{"b":"400000","o":"C90A85","s":"_ZN5mongo23WiredTigerIndexStandard8_unindexEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"C8F2F7","s":"_ZN5mongo15WiredTigerIndex7unindexEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"8ADE28","s":"_ZN5mongo17IndexAccessMethod6updateEPNS_16OperationContextERKNS_12UpdateTicketEPl"},{"b":"400000","o":"6D7384","s":"_ZN5mongo10Collection14updateDocumentEPNS_16OperationContextERKNS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS7_bbPNS_7OpDebugERNS_20oplogUpdateEntryArgsE"},{"b":"400000","o":"823611","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE"},{"b":"400000","o":"823FF8","s":"_ZN5mongo11UpdateStage4workEPm"},{"b":"400000","o":"A32A05","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"400000","o":"A330C9","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"400000","o":"A331C5","s":"_ZN5mongo12PlanExecutor11executePlanEv"},{"b":"400000","o":"93D569","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE"},{"b":"400000","o":"AC4374","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb"},{"b":"400000","o":"B5AC0E","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b"},{"b":"400000","o":"B54A10"},{"b":"400000","o":"B56656","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE"},{"b":"400000","o":"B57256","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"400000","o":"B5A55B","s":"_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE"},{"b":"400000","o":"ECE131","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"400000","o":"ECEA99","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"400000","o":"ECF5F0","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs"},{"b":"400000","o":"1769250","s":"execute_native_thread_routine"},{"b":"7F37CFF18000","o":"79D1"},{"b":"7F37CFB84000","o":"E88FD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "2.6.32-504.16.2.el6.x86_64", "version" : "#1 SMP Wed Apr 22 06:48:29 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "C3B1D60B7051B5D77B97AA9E4D263F39E90B2632" }, { "b" : "7FFF1D2FF000", "elfType" : 3, "buildId" : "044DC84B6E7F9EDDF8698512A1754429564EC692" }, { "b" : "7F37D0DBF000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7CE5058BE3F2AE6EBC0DF61E790C3FCED1A34C7B" }, { "b" : "7F37D09DB000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "096F60DC43484ECA364206125CED7D563FF9D027" }, { "b" : "7F37D07D3000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "583411D8786F86A1D6B8741C502831E6122445A7" }, { "b" : "7F37D05CF000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "454F8FC6CC6502C6401E5F9E221564D80665D277" }, { "b" : "7F37D034B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "C9A87F6A29ED1D3CB18F539845A45FE3A9877FF1" }, { "b" : "7F37D0135000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "9A6E4BDFA184364D81F7DFD789474C3FB8F98A00" }, { "b" : "7F37CFF18000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8DFF8E53D9F2B80C3C382E83EC17C828B536A39" }, { "b" : "7F37CFB84000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "1425CB3B4C2F49C8101ED9B8F1D289053B4DFA77" }, { "b" : "7F37D102C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F8E59B70E469F3A924A268911FF8FD0C37E7460" }, { "b" : "7F37CF940000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9B852585C66329AA02EFB28497E652A40F538E78" }, { "b" : "7F37CF659000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "EF3AACAFD6BF71BB861F194C1559153FB0B020E2" }, { "b" : "7F37CF455000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "152E2C18A7A2145021A8A879A01A82EE134E3946" }, { "b" : "7F37CF229000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "DDE6774979156442185836150FC0785170F8001F" }, { "b" : "7F37CF013000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7F37CEE08000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "A23DAFBCE170763BF1E836A8B26113F9CD20E0DA" }, { "b" : "7F37CEC05000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7F37CE9EB000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "58B696478044E028A5970D48A4ED50E164B43B36" }, { "b" : "7F37CE7CC000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13403e2]
 mongod(+0xF3F319) [0x133f319]
 mongod(+0xF3FB22) [0x133fb22]
 libpthread.so.0(+0xF710) [0x7f37cff27710]
 libc.so.6(gsignal+0x35) [0x7f37cfbb6625]
 libc.so.6(abort+0x175) [0x7f37cfbb7e05]
 mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12c3602]
 mongod(+0xCB3C63) [0x10b3c63]
 mongod(__wt_eventv+0x42C) [0x1aad81c]
 mongod(__wt_err+0x8D) [0x1aadc9d]
 mongod(__wt_panic+0x24) [0x1aae084]
 mongod(__wt_bm_read+0x82) [0x19ddff2]
 mongod(__wt_bt_read+0x1ED) [0x19fb50d]
 mongod(__wt_page_in_func+0x202) [0x1a00ab2]
 mongod(__wt_row_search+0x687) [0x1a20b27]
 mongod(__wt_btcur_remove+0x304) [0x19f2244]
 mongod(+0x163CC0A) [0x1a3cc0a]
 mongod(_ZN5mongo23WiredTigerIndexStandard8_unindexEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb+0xC5) [0x1090a85]
 mongod(_ZN5mongo15WiredTigerIndex7unindexEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0x77) [0x108f2f7]
 mongod(_ZN5mongo17IndexAccessMethod6updateEPNS_16OperationContextERKNS_12UpdateTicketEPl+0xB8) [0xcade28]
 mongod(_ZN5mongo10Collection14updateDocumentEPNS_16OperationContextERKNS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS7_bbPNS_7OpDebugERNS_20oplogUpdateEntryArgsE+0x984) [0xad7384]
 mongod(_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE+0xAD1) [0xc23611]
 mongod(_ZN5mongo11UpdateStage4workEPm+0x5D8) [0xc23ff8]
 mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe32a05]
 mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe330c9]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe331c5]
 mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x179) [0xd3d569]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0x924) [0xec4374]
 mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf5ac0e]
 mongod(+0xB54A10) [0xf54a10]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf56656]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf57256]
 mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf5a55b]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x12ce131]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12cea99]
 mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x12cf5f0]
 mongod(execute_native_thread_routine+0x20) [0x1b69250]
 libpthread.so.0(+0x79D1) [0x7f37cff1f9d1]
 libc.so.6(clone+0x6D) [0x7f37cfc6c8fd]
-----  END BACKTRACE  -----



 Comments   
Comment by Mark Agarunov [ 21/Mar/17 ]

Hello netwrom,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thanks,
Mark

Comment by Mark Agarunov [ 14/Mar/17 ]

Hello netwrom,

Thank you for the response. If possible, could you provide the complete logs from mongod when you attempt to start it? The errors and startup info in the logs may shed some light on the cause of this issue.

Thanks,
Mark

Comment by hu dai [ 14/Mar/17 ]

hi mark.agarunov

if MongoDB Crashed, I Can't start MongoDB Server,you method may be I can not use for me

Comment by Mark Agarunov [ 07/Mar/17 ]

Hello netwrom,

Please note that the method described in the linked blog post is not supported and may lead to further corruption in some circumstances. If you would like to attempt to repair the database, the steps would be as follows:

  • Find the name of the collection the corrupted index belongs to.

This can be found by running the following in the mongo shell and comparing the uri with the index that is logged, in this case index-84-9204097186848499981:

db.getCollectionNames().forEach(
  function(collection){
    var stats=db[collection].stats({indexDetails:true}).indexDetails; 
    for (var c in stats){
      print ("Collection: " + collection + ", Index: " + c + ", uri: " + stats[c].uri);
    }
  });

Thanks,
Mark

Comment by hu dai [ 07/Mar/17 ]

because I used qingclould service, maybe qingclould disk subsystem have problem

Comment by Alexander Gorrod [ 07/Mar/17 ]

This error indicates that there was a data corruption in your disk subsystem that corrupted the content of your database. When MongoDB encounters a data corruption, it shuts down immediately in order to avoid introducing more wide-spread corruption of data.

Our recommended resolution in this case is to:

  • Resync from another node in a replica set
  • Restore your database from a backup

Let us know if neither of those are options for you, and we will see if we can help further.

Generated at Thu Feb 08 04:17:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.