Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29970

mongodb database file WiredTiger.wt: encountered an illegal file format or internal value

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.14
    • Component/s: WiredTiger
    • Labels:
    • Linux
    • Hide

      power loss.

      Show
      power loss.

      Hi.
      After a unexpected power loss on my hostage provider my mongodb database was not working.
      I already tried the repair and remove locked files as suggested from my google searches, but didn't work.
      Here I leave the files for anyone who can help me repair this.
      thank you.

      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] MongoDB starting : pid=5694 port=27017 dbpath=/var/lib/mongodb 64-bit host=napalm
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] db version v3.2.14
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] git version: 92f6668a768ebf294bd4f494c50f48459198e6a3
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] modules: none
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] build environment:
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     distarch: x86_64
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-07-04T17:13:36.313+0100 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb" } }
      2017-07-04T17:13:36.337+0100 I -        [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-07-04T17:13:36.337+0100 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2017-07-04T17:13:36.337+0100 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (0) [1499184816:394726][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: read checksum error for 24576B block at offset 16384: block header checksum of 1814037912 doesn't match expected checksum of 1064729368
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (0) [1499184816:394789][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: WiredTiger.wt: encountered an illegal file format or internal value
      2017-07-04T17:13:36.394+0100 E STORAGE  [initandlisten] WiredTiger (-31804) [1499184816:394807][5694:0x7fc59e429c80], file:WiredTiger.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-07-04T17:13:36.394+0100 I -        [initandlisten] Fatal Assertion 28558
      2017-07-04T17:13:36.394+0100 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2017-07-04T17:13:36.412+0100 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x154e4f2 0x154d499 0x154dd02 0x7fc59d02f390 0x7fc59cc89428 0x7fc59cc8b02a 0x14cb733 0x126e2da 0x97b1e3 0x97b3c9 0x97b58f 0x1b94e95 0x1bae3cb 0x1bb5384 0x1bd5c40 0x1ba46d2 0x1bf19b4 0x1c7d4fe 0x1c16cca 0x1c7df98 0x1be85b7 0x1be0e35 0x1251e3a 0x124b1e5 0x114640e 0x9c7b31 0x97c35a 0x7fc59cc74830 0x9c2e29
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"114E4F2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"114D499"},{"b":"400000","o":"114DD02"},{"b":"7FC59D01E000","o":"11390"},{"b":"7FC59CC54000","o":"35428","s":"gsignal"},{"b":"7FC59CC54000","o":"3702A","s":"abort"},{"b":"400000","o":"10CB733","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E6E2DA"},{"b":"400000","o":"57B1E3","s":"__wt_eventv"},{"b":"400000","o":"57B3C9","s":"__wt_err"},{"b":"400000","o":"57B58F","s":"__wt_panic"},{"b":"400000","o":"1794E95","s":"__wt_bm_read"},{"b":"400000","o":"17AE3CB","s":"__wt_bt_read"},{"b":"400000","o":"17B5384","s":"__wt_page_in_func"},{"b":"400000","o":"17D5C40","s":"__wt_row_search"},{"b":"400000","o":"17A46D2","s":"__wt_btcur_insert"},{"b":"400000","o":"17F19B4"},{"b":"400000","o":"187D4FE"},{"b":"400000","o":"1816CCA","s":"__wt_log_scan"},{"b":"400000","o":"187DF98","s":"__wt_txn_recover"},{"b":"400000","o":"17E85B7","s":"__wt_connection_workers"},{"b":"400000","o":"17E0E35","s":"wiredtiger_open"},{"b":"400000","o":"E51E3A","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E4B1E5"},{"b":"400000","o":"D4640E","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5C7B31"},{"b":"400000","o":"57C35A","s":"main"},{"b":"7FC59CC54000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5C2E29","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.14", "gitVersion" : "92f6668a768ebf294bd4f494c50f48459198e6a3", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-81-generic", "version" : "#104-Ubuntu SMP Wed Jun 14 08:17:06 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B04D4C2514E2C891B5791D71A8F4246ECADF157D" }, { "b" : "7FFC51DD5000", "elfType" : 3, "buildId" : "324AC75434EC60709DEFA00F991DBCAB9848D1D5" }, { "b" : "7FC59DFAA000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FC59DB66000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FC59D95E000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FC59D75A000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FC59D451000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FC59D23B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FC59D01E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FC59CC54000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FC59E213000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x154e4f2]
       mongod(+0x114D499) [0x154d499]
       mongod(+0x114DD02) [0x154dd02]
       libpthread.so.0(+0x11390) [0x7fc59d02f390]
       libc.so.6(gsignal+0x38) [0x7fc59cc89428]
       libc.so.6(abort+0x16A) [0x7fc59cc8b02a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x14cb733]
       mongod(+0xE6E2DA) [0x126e2da]
       mongod(__wt_eventv+0x3BA) [0x97b1e3]
       mongod(__wt_err+0x8B) [0x97b3c9]
       mongod(__wt_panic+0x24) [0x97b58f]
       mongod(__wt_bm_read+0x115) [0x1b94e95]
       mongod(__wt_bt_read+0x1DB) [0x1bae3cb]
       mongod(__wt_page_in_func+0x11E4) [0x1bb5384]
       mongod(__wt_row_search+0x640) [0x1bd5c40]
       mongod(__wt_btcur_insert+0xB42) [0x1ba46d2]
       mongod(+0x17F19B4) [0x1bf19b4]
       mongod(+0x187D4FE) [0x1c7d4fe]
       mongod(__wt_log_scan+0xC4A) [0x1c16cca]
       mongod(__wt_txn_recover+0x348) [0x1c7df98]
       mongod(__wt_connection_workers+0x37) [0x1be85b7]
       mongod(wiredtiger_open+0x18F5) [0x1be0e35]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0xBDA) [0x1251e3a]
       mongod(+0xE4B1E5) [0x124b1e5]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3EE) [0x114640e]
       mongod(+0x5C7B31) [0x9c7b31]
       mongod(main+0x73A) [0x97c35a]
       libc.so.6(__libc_start_main+0xF0) [0x7fc59cc74830]
       mongod(_start+0x29) [0x9c2e29]
      -----  END BACKTRACE  -----
      
      

        1. _mdb_catalog.wt
          44 kB
        2. repair_attempt.tar.gz
          18 kB
        3. repair.log
          7 kB
        4. sizeStorer.wt
          36 kB
        5. storage.bson
          0.1 kB
        6. WiredTiger
          0.0 kB
        7. WiredTiger.lock
          0.0 kB
        8. WiredTiger.turtle
          1.0 kB
        9. WiredTiger.wt
          388 kB
        10. WiredTigerLAS.wt
          4 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            napalmd filipe silva
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: