Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29713

WiredTiger.wt is corrupted

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.7
    • Component/s: WiredTiger
    • Labels:
    • Linux

      After unclean shutdown my WiredTiger.wt file is corrupted. When I'm try to --repair I got this stacktrace:

      2017-06-19T19:32:57.073+0300 I CONTROL  [initandlisten] options: { repair: true, storage: { dbPath: ".", engine: "wiredTiger" } }
      2017-06-19T19:32:57.095+0300 W -        [initandlisten] Detected unclean shutdown - ./mongod.lock is not empty.
      2017-06-19T19:32:57.095+0300 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2017-06-19T19:32:57.095+0300 I STORAGE  [initandlisten] Detected WT journal files.  Running recovery from last checkpoint.
      2017-06-19T19:32:57.095+0300 I STORAGE  [initandlisten] journal to nojournal transition config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-06-19T19:32:57.102+0300 E STORAGE  [initandlisten] WiredTiger (0) [1497889977:102005][2433:0x7f1b31cc0c80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 3831956594 doesn't match expected checksum of 861916107
      2017-06-19T19:32:57.102+0300 E STORAGE  [initandlisten] WiredTiger (0) [1497889977:102056][2433:0x7f1b31cc0c80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2017-06-19T19:32:57.102+0300 E STORAGE  [initandlisten] WiredTiger (-31804) [1497889977:102066][2433:0x7f1b31cc0c80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-06-19T19:32:57.102+0300 I -        [initandlisten] Fatal Assertion 28558
      2017-06-19T19:32:57.102+0300 I -        [initandlisten]
      
      ***aborting after fassert() failure
      
      
      2017-06-19T19:32:57.118+0300 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x15147a2 0x1513749 0x1513fb2 0x7f1b308c6390 0x7f1b30521428 0x7f1b3052302a 0x149c0d3 0x1241992 0x1c23417 0x1c2384e 0x1c23c54 0x1b550af 0x1b5561b 0x1b52558 0x1b56617 0x1b73608 0x1ba75d0 0x1c2201b 0x1c225b0 0x1c2272c 0x1bb6278 0x1c1f258 0x1be94ef 0x1be95ac 0x1ba42cc 0x1226eb0 0x1220495 0x111da78 0x9be8c0 0x974322 0x7f1b3050c830 0x9b8bf9
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"11147A2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"1113749"},{"b":"400000","o":"1113FB2"},{"b":"7F1B308B5000","o":"11390"},{"b":"7F1B304EC000","o":"35428","s":"gsignal"},{"b":"7F1B304EC000","o":"3702A","s":"abort"},{"b":"400000","o":"109C0D3","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"E41992"},{"b":"400000","o":"1823417","s":"__wt_eventv"},{"b":"400000","o":"182384E","s":"__wt_err"},{"b":"400000","o":"1823C54","s":"__wt_panic"},{"b":"400000","o":"17550AF","s":"__wt_block_extlist_read"},{"b":"400000","o":"175561B","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"1752558","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"1756617"},{"b":"400000","o":"1773608","s":"__wt_btree_open"},{"b":"400000","o":"17A75D0","s":"__wt_conn_btree_open"},{"b":"400000","o":"182201B","s":"__wt_session_get_btree"},{"b":"400000","o":"18225B0","s":"__wt_session_get_btree"},{"b":"400000","o":"182272C","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"17B6278","s":"__wt_curfile_open"},{"b":"400000","o":"181F258"},{"b":"400000","o":"17E94EF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"17E95AC","s":"__wt_metadata_cursor"},{"b":"400000","o":"17A42CC","s":"wiredtiger_open"},{"b":"400000","o":"E26EB0","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb"},{"b":"400000","o":"E20495"},{"b":"400000","o":"D1DA78","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5BE8C0","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"574322","s":"main"},{"b":"7F1B304EC000","o":"20830","s":"__libc_start_main"},{"b":"400000","o":"5B8BF9","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.2.7", "gitVersion" : "4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-79-generic", "version" : "#100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B025CB0CF792AB652E442FF480378B48601044BE" }, { "b" : "7FFFFD1F1000", "elfType" : 3, "buildId" : "9446C0DD55430C6F14109300BF1799D05C90D64D" }, { "b" : "7F1B31841000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7F1B313FD000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7F1B311F5000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "0DBB8C21FC5D977098CA718BA2BFD6C4C21172E9" }, { "b" : "7F1B30FF1000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C0C5B7F18348654040534B050B110D32A19EA38D" }, { "b" : "7F1B30CE8000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "05451CB4D66C321691F64F253880B7CE5B8812A6" }, { "b" : "7F1B30AD2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F1B308B5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "84538E3C6CFCD5D4E3C0D2B6C3373F802915A498" }, { "b" : "7F1B304EC000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CBFA941A8EB7A11E4F90E81B66FCD5A820995D7C" }, { "b" : "7F1B31AAA000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A7D5A820B802049276B1FC26C8E845A3E194EB6B" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x15147a2]
       mongod(+0x1113749) [0x1513749]
       mongod(+0x1113FB2) [0x1513fb2]
       libpthread.so.0(+0x11390) [0x7f1b308c6390]
       libc.so.6(gsignal+0x38) [0x7f1b30521428]
       libc.so.6(abort+0x16A) [0x7f1b3052302a]
       mongod(_ZN5mongo13fassertFailedEi+0x93) [0x149c0d3]
       mongod(+0xE41992) [0x1241992]
       mongod(__wt_eventv+0x4F7) [0x1c23417]
       mongod(__wt_err+0x8E) [0x1c2384e]
       mongod(__wt_panic+0x24) [0x1c23c54]
       mongod(__wt_block_extlist_read+0x6F) [0x1b550af]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x1b5561b]
       mongod(__wt_block_checkpoint_load+0x1F8) [0x1b52558]
       mongod(+0x1756617) [0x1b56617]
       mongod(__wt_btree_open+0xBB8) [0x1b73608]
       mongod(__wt_conn_btree_open+0x140) [0x1ba75d0]
       mongod(__wt_session_get_btree+0xEB) [0x1c2201b]
       mongod(__wt_session_get_btree+0x680) [0x1c225b0]
       mongod(__wt_session_get_btree_ckpt+0x12C) [0x1c2272c]
       mongod(__wt_curfile_open+0x218) [0x1bb6278]
       mongod(+0x181F258) [0x1c1f258]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1be94ef]
       mongod(__wt_metadata_cursor+0x3C) [0x1be95ac]
       mongod(wiredtiger_open+0x188C) [0x1ba42cc]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_S8_mbbb+0x820) [0x1226eb0]
       mongod(+0xE20495) [0x1220495]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x3E8) [0x111da78]
       mongod(_ZN5mongo13initAndListenEi+0x480) [0x9be8c0]
       mongod(main+0x752) [0x974322]
       libc.so.6(__libc_start_main+0xF0) [0x7f1b3050c830]
       mongod(_start+0x29) [0x9b8bf9]
      -----  END BACKTRACE  -----
      

      My mongodb version: v3.2.7 (4249c1d2b5999ebbf1fdf3bc0e0e3b3ff5c0aaf2)

      I'm attached WiredTiger files. Please, could you help?

        1. repair_attempt.tar.gz
          37 kB
        2. WiredTiger
          0.0 kB
        3. WiredTiger.turtle
          0.9 kB
        4. WiredTiger.wt
          648 kB
        5. WiredTigerLAS.wt
          4 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            a.maslov Alexander Maslov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: