Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-49319

WiredTiger crash after shutdown

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Duplicate
    • None
    • None
    • None
    • None
    • ALL

    Description

      Hello! I have issue with starting mongodb 4 after sudder shutdown of server.

      I have have only one large singe node mongodb without backup.
      I'm not sure if it correct to post it here

       Appreciate any help.

      mongod --repair

      log:

       

      2020-07-05T14:55:04.120+0000 E STORAGE [initandlisten] WiredTiger error (0) [1593960904:120370][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_bm_corrupt_dump, 135: {24331131842560, 188416, 969740655}: (chunk 184 of 184): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Raw: [1593960904:120370][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_bm_corrupt_dump, 135: {24331131842560, 188416, 969740655}: (chunk 184 of 184): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
       2020-07-05T14:55:04.120+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1593960904:120454][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_block_read_off, 281: collection-5-7105085878558723654.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1593960904:120454][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_block_read_off, 281: collection-5-7105085878558723654.wt: fatal read error: WT_ERROR: non-specific WiredTiger error
       2020-07-05T14:55:04.120+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1593960904:120480][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1593960904:120480][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, WT_CURSOR.insert: __wt_panic, 494: the process must exit and restart: WT_PANIC: WiredTiger library panic
       2020-07-05T14:55:04.120+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1593960904:120504][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, txn-recover: __txn_op_apply, 287: operation apply failed during recovery: operation type 4 at LSN 268058/40238976: WT_PANIC: WiredTiger library panic Raw: [1593960904:120504][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, txn-recover: __txn_op_apply, 287: operation apply failed during recovery: operation type 4 at LSN 268058/40238976: WT_PANIC: WiredTiger library panic
       2020-07-05T14:55:04.120+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1593960904:120551][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, txn-recover: __wt_txn_recover, 706: Recovery failed: WT_PANIC: WiredTiger library panic Raw: [1593960904:120551][1:0x7f9d11fc6a80], file:collection-5-7105085878558723654.wt, txn-recover: __wt_txn_recover, 706: Recovery failed: WT_PANIC: WiredTiger library panic
       2020-07-05T14:55:04.121+0000 E STORAGE [initandlisten] WiredTiger error (0) [1593960904:121027][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 350: cache server: exiting with 18 pages in memory and 9 pages evicted Raw: [1593960904:121027][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 350: cache server: exiting with 18 pages in memory and 9 pages evicted
       2020-07-05T14:55:04.121+0000 E STORAGE [initandlisten] WiredTiger error (0) [1593960904:121066][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 355: cache server: exiting with 79216 image bytes in memory Raw: [1593960904:121066][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 355: cache server: exiting with 79216 image bytes in memory
       2020-07-05T14:55:04.121+0000 E STORAGE [initandlisten] WiredTiger error (0) [1593960904:121085][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 358: cache server: exiting with 131776 bytes in memory Raw: [1593960904:121085][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 358: cache server: exiting with 131776 bytes in memory
       2020-07-05T14:55:04.121+0000 E STORAGE [initandlisten] WiredTiger error (0) [1593960904:121100][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 364: cache server: exiting with 83777 bytes dirty and 4 pages dirty Raw: [1593960904:121100][1:0x7f9d11fc6a80], connection: __wt_cache_destroy, 364: cache server: exiting with 83777 bytes dirty and 4 pages dirty
       2020-07-05T14:55:04.122+0000 F STORAGE [initandlisten] Failed to salvage WiredTiger metadata: -31809: WT_TRY_SALVAGE: database corruption detected
       2020-07-05T14:55:04.122+0000 F - [initandlisten] Fatal Assertion 50947 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 722
       2020-07-05T14:55:04.122+0000 F - [initandlisten]
      ***aborting after fassert() failure
       
      

       

      Attachments

        1. WiredTiger.wt
          164 kB
        2. WiredTiger.turtle
          1.0 kB
        3. WiredTiger
          0.0 kB
        4. repair-1.log
          4.65 MB
        5. image-2020-07-10-14-46-57-804.png
          image-2020-07-10-14-46-57-804.png
          187 kB
        6. image-2020-07-10-14-45-44-647.png
          image-2020-07-10-14-45-44-647.png
          216 kB
        7. image-2020-07-10-14-43-27-233.png
          image-2020-07-10-14-43-27-233.png
          72 kB

        Issue Links

          Activity

            People

              dmitry.agranat@mongodb.com Dmitry Agranat
              ginn.tar.gz@gmail.com Anton Zolotukhin
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: