Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29970

mongodb database file WiredTiger.wt: encountered an illegal file format or internal value

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.14
    • Component/s: WiredTiger
    • Linux
    • Hide

      power loss.

      Show
      power loss.

      Hi.
      After a unexpected power loss on my hostage provider my mongodb database was not working.
      I already tried the repair and remove locked files as suggested from my google searches, but didn't work.
      Here I leave the files for anyone who can help me repair this.
      thank you.

      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] MongoDB starting : pid=5694 port=27017 dbpath=/var/lib/mongodb 64-bit host=napalm
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] db version v3.2.14
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] git version: 92f6668a768ebf294bd4f494c50f48459198e6a3
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] modules: none
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] build environment:
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     distarch: x86_64
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
      2017-07-04T17:13:36.337+0100 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-07-04T17:13:36.337+0100 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2017-07-04T17:13:36.337+0100 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (0) [1499184816:394726][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 24576B block at offset 16384: block header checksum of 1814037912 doesn't match expected checksum of 1064729368
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (0) [1499184816:394789][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (-31804) [1499184816:394807][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-07-04T17:13:36.394+0100 I -        [initandlisten] Fatal Assertion 28558
      2017-07-04T17:13:36.394+0100 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2017-07-04T17:13:36.412+0100 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x154e4f2 0x154d499 0x154dd02 0x7fc59d02f390 0x7fc59cc89428 0x7fc59cc8b02a 0x14cb733 0x126e2da 0x97b1e3 0x97b3c9 0x97b58f 0x1b94e95 0x1bae3cb 0x1bb5384 0x1bd5c40 0x1ba46d2 0x1bf19b4 0x1c7d4fe 0x1c16cca 0x1c7df98 0x1be85b7 0x1be0e35 0x1251e3a 0x124b1e5 0x114640e 0x9c7b31 0x97c35a 0x7fc59cc74830 0x9c2e29
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"114E4F2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"114D499"},{"b":"400000","o":"114DD02"},{"b":"7FC59D01E000","o":"11390"},{"b":"7FC59CC54000","o":"35428","s":"gsignal"},{"b":"7FC59CC54000","o":"3702A","s":"abort"},{"b":"400000","o":"10CB733","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E6E2DA"},{"b":"400000","o":"57B1E3","s":"__wt_eventv"},{"b":"400000","o":"57B3C9","s":"__wt_err"},{"b":"400000","o":"57B58F","s":"__wt_panic"},{"b":"400000","o":"1794E95","s":"__wt_bm_read"},{"b":"400000","o":"17AE3CB","s":"__wt_bt_read"},{"b":"400000","o":"17B5384","s":"__wt_page_in_func"},{"b":"400000","o":"17D5C40","s":"__wt_row_search"},{"b":"400000","o":"17A46D2","s":"__wt_btcur_insert"},{"b":"400000","o":"17F19B4"},{"b":"400000","o":"187D4FE"},{"b":"400000","o":"1816CCA","s":"__wt_log_scan"},{"b":"400000","o":"187DF98","s":"__wt_txn_recover"},{"b":"400000","o":"17E85B7","s":"__wt_connection_workers"},{"b":"400000","o":"17E0E35","s":"wiredtiger_open"},{"b":"400000","o":"E51E3A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E4B1E5"},{"b":"400000","o":"D4640E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5C7B31"},{"b":"400000","o":"57C35A","s":"main"},{"b":"7FC59CC54000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5C2E29","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.14", "gitVersion" : "92f6668a768ebf294bd4f494c50f48459198e6a3", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-81-generic", "version" : "#104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B04D4C2514E2C891B5791D71A8F4246ECADF157D" }, { "b" : "7FFC51DD5000", "elfType" : 3, "buildId" : "324AC75434EC60709DEFA00F991DBCAB9848D1D5" }, { "b" : "7FC59DFAA000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FC59DB66000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FC59D95E000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FC59D75A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FC59D451000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FC59D23B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FC59D01E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FC59CC54000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FC59E213000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x154e4f2]
       mongod(+0x114D499) [0x154d499]
       mongod(+0x114DD02) [0x154dd02]
       libpthread.so.0(+0x11390) [0x7fc59d02f390]
       libc.so.6(gsignal+0x38) [0x7fc59cc89428]
       libc.so.6(abort+0x16A) [0x7fc59cc8b02a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14cb733]
       mongod(+0xE6E2DA) [0x126e2da]
       mongod(__wt_eventv+0x3BA) [0x97b1e3]
       mongod(__wt_err+0x8B) [0x97b3c9]
       mongod(__wt_panic+0x24) [0x97b58f]
       mongod(__wt_bm_read+0x115) [0x1b94e95]
       mongod(__wt_bt_read+0x1DB) [0x1bae3cb]
       mongod(__wt_page_in_func+0x11E4) [0x1bb5384]
       mongod(__wt_row_search+0x640) [0x1bd5c40]
       mongod(__wt_btcur_insert+0xB42) [0x1ba46d2]
       mongod(+0x17F19B4) [0x1bf19b4]
       mongod(+0x187D4FE) [0x1c7d4fe]
       mongod(__wt_log_scan+0xC4A) [0x1c16cca]
       mongod(__wt_txn_recover+0x348) [0x1c7df98]
       mongod(__wt_connection_workers+0x37) [0x1be85b7]
       mongod(wiredtiger_open+0x18F5) [0x1be0e35]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xBDA) [0x1251e3a]
       mongod(+0xE4B1E5) [0x124b1e5]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x114640e]
       mongod(+0x5C7B31) [0x9c7b31]
       mongod(main+0x73A) [0x97c35a]
       libc.so.6(__libc_start_main+0xF0) [0x7fc59cc74830]
       mongod(_start+0x29) [0x9c2e29]
      -----  END BACKTRACE  -----
      
      

        1. _mdb_catalog.wt
          44 kB
          filipe silva
        2. repair_attempt.tar.gz
          18 kB
          Kelsey Schubert
        3. repair.log
          7 kB
          filipe silva
        4. sizeStorer.wt
          36 kB
          filipe silva
        5. storage.bson
          0.1 kB
          filipe silva
        6. WiredTiger
          0.0 kB
          filipe silva
        7. WiredTiger.lock
          0.0 kB
          filipe silva
        8. WiredTiger.turtle
          1.0 kB
          filipe silva
        9. WiredTiger.wt
          388 kB
          filipe silva
        10. WiredTigerLAS.wt
          4 kB
          filipe silva

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            napalmd filipe silva
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: