Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-27538

WiredTiger.wt corrupt - unable to repair to start mongo

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.8
    • Component/s: WiredTiger
    • None
    • Environment:
      Linux 14.04

      My DB is corrupted and unusable after an intensive memory and CPU process ran for ~20 minutes. I've uploaded my WiredTiger.turle & WiredTiger.wt files below... any help is appreciated!

      admin@jenkins:/data/db$ sudo mongod
      2016-12-28T21:16:40.638-0800 I CONTROL  [initandlisten] MongoDB starting : pid=16897 port=27017 dbpath=/data/db 64-bit host=jenkins
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] db version v3.2.8
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] allocator: tcmalloc
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] modules: none
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] build environment:
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     distmod: ubuntu1404
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     distarch: x86_64
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten]     target_arch: x86_64
      2016-12-28T21:16:40.639-0800 I CONTROL  [initandlisten] options: {}
      2016-12-28T21:16:40.668-0800 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2016-12-28T21:16:40.668-0800 W -        [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
      2016-12-28T21:16:40.668-0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2016-12-28T21:16:40.668-0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (0) [1482988600:686354][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 57344: block header checksum of 3078277486 doesn't match expected checksum of 3500840698
      2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (0) [1482988600:686594][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2016-12-28T21:16:40.686-0800 E STORAGE  [initandlisten] WiredTiger (-31804) [1482988600:686705][16897:0x7f7ca3c01cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2016-12-28T21:16:40.686-0800 I -        [initandlisten] Fatal Assertion 28558
      2016-12-28T21:16:40.686-0800 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2016-12-28T21:16:40.709-0800 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x131ce72 0x131bfc9 0x131c7d2 0x7f7ca287d340 0x7f7ca24decc9 0x7f7ca24e20d8 0x12a6772 0x10a0923 0x1a7e31c 0x1a7e7dd 0x1a7ebc4 0x19affac 0x19b0523 0x19ad4b5 0x19b1569 0x19cdffc 0x1a04270 0x1a7d047 0x1a7d589 0x1a7d6ab 0x1a12858 0x1a7a1d5 0x1a44ccf 0x1a44dce 0x1a00e11 0x10889cf 0x1084c93 0xfadc48 0x9b48fd 0x96e04d 0x7f7ca24c9ec5 0x9b1037
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F1CE72","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F1BFC9"},{"b":"400000","o":"F1C7D2"},{"b":"7F7CA286D000","o":"10340"},{"b":"7F7CA24A8000","o":"36CC9","s":"gsignal"},{"b":"7F7CA24A8000","o":"3A0D8","s":"abort"},{"b":"400000","o":"EA6772","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CA0923"},{"b":"400000","o":"167E31C","s":"__wt_eventv"},{"b":"400000","o":"167E7DD","s":"__wt_err"},{"b":"400000","o":"167EBC4","s":"__wt_panic"},{"b":"400000","o":"15AFFAC","s":"__wt_block_extlist_read"},{"b":"400000","o":"15B0523","s":"__wt_block_extlist_read_avail"},{"b":"400000","o":"15AD4B5","s":"__wt_block_checkpoint_load"},{"b":"400000","o":"15B1569"},{"b":"400000","o":"15CDFFC","s":"__wt_btree_open"},{"b":"400000","o":"1604270","s":"__wt_conn_btree_open"},{"b":"400000","o":"167D047","s":"__wt_session_get_btree"},{"b":"400000","o":"167D589","s":"__wt_session_get_btree"},{"b":"400000","o":"167D6AB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1612858","s":"__wt_curfile_open"},{"b":"400000","o":"167A1D5"},{"b":"400000","o":"1644CCF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1644DCE","s":"__wt_metadata_cursor"},{"b":"400000","o":"1600E11","s":"wiredtiger_open"},{"b":"400000","o":"C889CF","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C84C93"},{"b":"400000","o":"BADC48","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B48FD","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"56E04D","s":"main"},{"b":"7F7CA24A8000","o":"21EC5","s":"__libc_start_main"},{"b":"400000","o":"5B1037"}],"processInfo":{ "mongodbVersion" : "3.2.8", "gitVersion" : "ed70e33130c977bda0024c125b56d159573dbaf0", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-71-generic", "version" : "#114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "A53FF676E1D627BD1D9B1BF524DEFA13B667EE83" }, { "b" : "7FFFF2EEF000", "elfType" : 3, "buildId" : "C2B4C020D20EE489936A4B11AF2C695FD0ECD484" }, { "b" : "7F7CA378E000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "D08DD65F97859C71BB2CBBF1043BD968EFE18AAD" }, { "b" : "7F7CA33B3000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "F86FA9FB4ECEB4E06B40DBDF761A4172B70A4229" }, { "b" : "7F7CA31AB000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7F7CA2FA7000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7F7CA2CA1000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7F7CA2A8B000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7F7CA286D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7F7CA24A8000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7F7CA39ED000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x131ce72]
       mongod(+0xF1BFC9) [0x131bfc9]
       mongod(+0xF1C7D2) [0x131c7d2]
       libpthread.so.0(+0x10340) [0x7f7ca287d340]
       libc.so.6(gsignal+0x39) [0x7f7ca24decc9]
       libc.so.6(abort+0x148) [0x7f7ca24e20d8]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12a6772]
       mongod(+0xCA0923) [0x10a0923]
       mongod(__wt_eventv+0x42C) [0x1a7e31c]
       mongod(__wt_err+0x8D) [0x1a7e7dd]
       mongod(__wt_panic+0x24) [0x1a7ebc4]
       mongod(__wt_block_extlist_read+0x6C) [0x19affac]
       mongod(__wt_block_extlist_read_avail+0x33) [0x19b0523]
       mongod(__wt_block_checkpoint_load+0x3C5) [0x19ad4b5]
       mongod(+0x15B1569) [0x19b1569]
       mongod(__wt_btree_open+0xC7C) [0x19cdffc]
       mongod(__wt_conn_btree_open+0x140) [0x1a04270]
       mongod(__wt_session_get_btree+0xE7) [0x1a7d047]
       mongod(__wt_session_get_btree+0x629) [0x1a7d589]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1a7d6ab]
       mongod(__wt_curfile_open+0x218) [0x1a12858]
       mongod(+0x167A1D5) [0x1a7a1d5]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a44ccf]
       mongod(__wt_metadata_cursor+0x7E) [0x1a44dce]
       mongod(wiredtiger_open+0x1541) [0x1a00e11]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x10889cf]
       mongod(+0xC84C93) [0x1084c93]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfadc48]
       mongod(_ZN5mongo13initAndListenEi+0x3DD) [0x9b48fd]
       mongod(main+0x15D) [0x96e04d]
       libc.so.6(__libc_start_main+0xF5) [0x7f7ca24c9ec5]
       mongod(+0x5B1037) [0x9b1037]
      -----  END BACKTRACE  -----
      

        1. repair_attempt.tar.gz
          6 kB
        2. WiredTiger.turtle
          0.9 kB
        3. WiredTiger.wt
          144 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            coryrobinson42@gmail.com Cory Robinson
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: