Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26924

Cannot start or --repair mongod because of unclean shutdown (due to running out of disk space)

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.9
    • Component/s: WiredTiger
    • None
    • ALL

      I have a small 1.7GB database that was running fine, but the drive it was on got filled up and so mongod crashed. Subsequently it's impossible to start mongod or to --repair, with the following error:

      $ mongod --dbpath db --repair
      2016-11-06T23:50:43.427-0500 I CONTROL  [initandlisten] MongoDB starting : pid=15289 port=27017 dbpath=db 64-bit host=cryoem
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] db version v3.2.9
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] allocator: tcmalloc
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] modules: none
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] build environment:
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten]     distarch: x86_64
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten]     target_arch: x86_64
      2016-11-06T23:50:43.432-0500 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "db" } }
      2016-11-06T23:50:43.518-0500 I -        [initandlisten] Detected data files in db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2016-11-06T23:50:43.518-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=37G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
      2016-11-06T23:50:43.535-0500 E STORAGE  [initandlisten] WiredTiger (0) [1478494243:535100][15289:0x7f652cdcccc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 1970325808 doesn't match expected checksum of 3338308183
      2016-11-06T23:50:43.535-0500 E STORAGE  [initandlisten] WiredTiger (0) [1478494243:535188][15289:0x7f652cdcccc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2016-11-06T23:50:43.535-0500 E STORAGE  [initandlisten] WiredTiger (-31804) [1478494243:535214][15289:0x7f652cdcccc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2016-11-06T23:50:43.535-0500 I -        [initandlisten] Fatal Assertion 28558
      2016-11-06T23:50:43.535-0500 I -        [initandlisten]
      
      ***aborting after fassert() failure
      
      
      2016-11-06T23:50:43.573-0500 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x1307192 0x13060b9 0x13068c2 0x7f652c097330 0x7f652bcf8c37 0x7f652bcfc028 0x1299422 0x108a733 0x1a7e13c 0x1a7e5bd 0x1a7e9a4 0x19a9c3c 0x19aa193 0x19a6e65 0x19ab1f9 0x19c888c 0x1a00700 0x1a7d087 0x1a7d5c9 0x1a7d6eb 0x1a0f2e8 0x1a7a225 0x1a43f0f 0x1a4400e 0x19fc4e3 0x1071baf 0x106dcf3 0xf91de8 0x95f6fe 0x961c90 0x964ead 0x7f652bce3f45 0x95c0b9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F07192","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F060B9"},{"b":"400000","o":"F068C2"},{"b":"7F652C087000","o":"10330"},{"b":"7F652BCC2000","o":"36C37","s":"gsignal"},{"b":"7F652BCC2000","o":"3A028","s":"abort"},{"b":"400000","o":"E99422","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C8A733"},{"b":"400000","o":"167E13C","s":"__wt_eventv"},{"b":"400000","o":"167E5BD","s":"__wt_err"},{"b":"400000","o":"167E9A4","s":"__wt_panic"},{"b":"400000","o":"15A9C3C","s":"__wt_block_extlist_read"},{"b":"400000","o":"15AA193","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15A6E65","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15AB1F9"},{"b":"400000","o":"15C888C","s":"__wt_btree_open"},{"b":"400000","o":"1600700","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D087","s":"__wt_session_get_btree"},{"b":"400000","o":"167D5C9","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6EB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"160F2E8","s":"__wt_curfile_open"},{"b":"400000","o":"167A225"},{"b":"400000","o":"1643F0F","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"164400E","s":"__wt_metadata_cursor"},{"b":"400000","o":"15FC4E3","s":"wiredtiger_open"},{"b":"400000","o":"C71BAF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C6DCF3"},{"b":"400000","o":"B91DE8","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"55F6FE"},{"b":"400000","o":"561C90","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"564EAD","s":"main"},{"b":"7F652BCC2000","o":"21F45","s":"__libc_start_main"},{"b":"400000","o":"55C0B9"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.16.0-76-generic", "version" : "#98~14.04.1-Ubuntu SMP Fri Jun 24 17:04:54 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFFB59F4000", "elfType" : 3 }, { "b" : "7F652C9C5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F652C7C1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F652C4BB000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F652C2A5000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F652C087000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F652BCC2000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F652CBCD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1307192]
       mongod(+0xF060B9) [0x13060b9]
       mongod(+0xF068C2) [0x13068c2]
       libpthread.so.0(+0x10330) [0x7f652c097330]
       libc.so.6(gsignal+0x37) [0x7f652bcf8c37]
       libc.so.6(abort+0x148) [0x7f652bcfc028]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x1299422]
       mongod(+0xC8A733) [0x108a733]
       mongod(__wt_eventv+0x42C) [0x1a7e13c]
       mongod(__wt_err+0x8D) [0x1a7e5bd]
       mongod(__wt_panic+0x24) [0x1a7e9a4]
       mongod(__wt_block_extlist_read+0x6C) [0x19a9c3c]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19aa193]
       mongod(__wt_block_checkpoint_load+0x3C5) [0x19a6e65]
       mongod(+0x15AB1F9) [0x19ab1f9]
       mongod(__wt_btree_open+0xC7C) [0x19c888c]
       mongod(__wt_conn_btree_open+0x140) [0x1a00700]
       mongod(__wt_session_get_btree+0xE7) [0x1a7d087]
       mongod(__wt_session_get_btree+0x629) [0x1a7d5c9]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6eb]
       mongod(__wt_curfile_open+0x218) [0x1a0f2e8]
       mongod(+0x167A225) [0x1a7a225]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a43f0f]
       mongod(__wt_metadata_cursor+0x7E) [0x1a4400e]
       mongod(wiredtiger_open+0x1433) [0x19fc4e3]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x1071baf]
       mongod(+0xC6DCF3) [0x106dcf3]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xf91de8]
       mongod(+0x55F6FE) [0x95f6fe]
       mongod(_ZN5mongo13initAndListenEi+0x10) [0x961c90]
       mongod(main+0x15D) [0x964ead]
       libc.so.6(__libc_start_main+0xF5) [0x7f652bce3f45]
       mongod(+0x55C0B9) [0x95c0b9]
      -----  END BACKTRACE  -----
      Aborted (core dumped)
      

      I believe this means the WiredTiger.wt file is somehow corrupt, seems like there are a lot of other tickets like this and some kind of manual repair needs to be done. I'm attaching all the relavant non-collection/index files from the db, hopefully someone can help!
      Thanks in advance.

        1. WiredTigerLAS.wt
          4 kB
        2. WiredTiger.wt
          188 kB
        3. WiredTiger.turtle
          0.9 kB
        4. WiredTiger.lock
          0.0 kB
        5. WiredTiger
          0.0 kB
        6. sizeStorer.wt
          36 kB
        7. mongorepair.py
          5 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            alipunjani Ali Punjani
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: