Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41420

repair after crash due to a full storage

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.4
    • Component/s: Admin, Build
    • Labels:
    • ALL
    • Hide

      mongod --dbpath /var/lib/mongodb --repair 

      Show
      mongod --dbpath /var/lib/mongodb --repair 

      Hi, 
      I have an issue on mongod.
      One day ago , my hard disk  storage  was full so mongodb shut down. 
      I add Go on my hard disk and now when i want to restart my mongo , it doesn't work . 

      I m on unbutu 16 and i tryied to do that : mongod --dbpath /var/lib/mongodb --repair 
      but i have systematically, after a while , this error: 

      // 2019-05-31T14:13:25.375+0000 E STORAGE  [initandlisten] WiredTiger error (-31802) [1559312005:375886][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, WT_CURSOR.insert: __wt_block_read_off, 302: collection-2818--1460086324816199596.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1559312005:375886][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, WT_CURSOR.insert: __wt_block_read_off, 302: collection-2818--1460086324816199596.wt: fatal read error: WT_ERROR: non-specific WiredTiger error
2019-05-31T14:13:25.375+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1559312005:375912][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, WT_CURSOR.insert: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1559312005:375912][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, WT_CURSOR.insert: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-05-31T14:13:25.375+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1559312005:375939][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, txn-recover: __txn_op_apply, 281: operation apply failed during recovery: operation type 4 at LSN 225/73807232: WT_PANIC: WiredTiger library panic Raw: [1559312005:375939][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, txn-recover: __txn_op_apply, 281: operation apply failed during recovery: operation type 4 at LSN 225/73807232: WT_PANIC: WiredTiger library panic
2019-05-31T14:13:25.376+0000 E STORAGE  [initandlisten] WiredTiger error (-31804) [1559312005:375995][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, txn-recover: __wt_txn_recover, 740: Recovery failed: WT_PANIC: WiredTiger library panic Raw: [1559312005:375995][8390:0x7fddddd90a40], file:collection-2818--1460086324816199596.wt, txn-recover: __wt_txn_recover, 740: Recovery failed: WT_PANIC: WiredTiger library panic
2019-05-31T14:13:25.383+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1559312005:383068][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 384: cache server: exiting with 7 pages in memory and 0 pages evicted Raw: [1559312005:383068][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 384: cache server: exiting with 7 pages in memory and 0 pages evicted
2019-05-31T14:13:25.383+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1559312005:383126][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 389: cache server: exiting with 149858 image bytes in memory Raw: [1559312005:383126][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 389: cache server: exiting with 149858 image bytes in memory
2019-05-31T14:13:25.383+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1559312005:383146][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 393: cache server: exiting with 154498 bytes in memory Raw: [1559312005:383146][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 393: cache server: exiting with 154498 bytes in memory
2019-05-31T14:13:25.383+0000 E STORAGE  [initandlisten] WiredTiger error (0) [1559312005:383163][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 400: cache server: exiting with 32496 bytes dirty and 1 pages dirty Raw: [1559312005:383163][8390:0x7fddddd90a40], connection: __wt_cache_destroy, 400: cache server: exiting with 32496 bytes dirty and 1 pages dirty
2019-05-31T14:13:25.388+0000 E -        [initandlisten] Assertion: Location28718: -31809: WT_TRY_SALVAGE: database corruption detected src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 504
2019-05-31T14:13:25.396+0000 I STORAGE  [initandlisten] exception in initAndListen: Location28718: -31809: WT_TRY_SALVAGE: database corruption detected, terminating
2019-05-31T14:13:25.406+0000 I CONTROL  [initandlisten] now exiting
2019-05-31T14:13:25.406+0000 I CONTROL  [initandlisten] shutting down with code:100

      I tryied to do that more than once , but still the same issue.  Even when i add --nojournal it doesn't work
      I removed mongod.lock for that. 
      Do you have any solution to resolved it?

      Thanks a lot 
      Thomas

       

       

        1. log_repair.log
          504 kB
        2. log_after_repair_failed.log
          6 kB
        3. crash_mongo_first_error.log
          333 kB

            Assignee:
            eric.sedor@mongodb.com Eric Sedor
            Reporter:
            tigrou34 thomas poidevin
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: