Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34083

WiredTiger checksum error

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.6
    • Component/s: WiredTiger
    • Labels:
    • Linux

      Hi,

      I have a problem.
      One day, my mongodb environment encountered suddenly power off.
      After that, I can't start mongod, and following messages displayed.
      Could you help me to recover my database, please?

      2018-03-19T09:41:44.979+0900 I CONTROL  [initandlisten] MongoDB starting : pid=32098 port=27017 dbpath=/data/db 64-bit host=mangodb-xiao
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] db version v3.4.6
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] git version: c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] allocator: tcmalloc
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] modules: enterprise 
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] build environment:
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten]     distmod: ubuntu1604
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten]     distarch: x86_64
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten]     target_arch: x86_64
      2018-03-19T09:41:44.983+0900 I CONTROL  [initandlisten] options: {}
      2018-03-19T09:41:45.036+0900 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2018-03-19T09:41:45.038+0900 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=7511M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2018-03-19T09:41:45.052+0900 E STORAGE  [initandlisten] WiredTiger error (0) [1521420105:52354][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 49152: block header checksum of 3126403452 doesn't match expected checksum of 71792573
      2018-03-19T09:41:45.052+0900 E STORAGE  [initandlisten] WiredTiger error (0) [1521420105:52407][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2018-03-19T09:41:45.052+0900 E STORAGE  [initandlisten] WiredTiger error (-31804) [1521420105:52415][32098:0x7ff0afbdcd40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2018-03-19T09:41:45.052+0900 I -        [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      2018-03-19T09:41:45.052+0900 I -        [initandlisten] 
      
      ***aborting after fassert() failure
      
      
      2018-03-19T09:41:45.090+0900 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x55b0bb686df1 0x55b0bb686009 0x55b0bb6864ed 0x7ff0acd06390 0x7ff0ac961428 0x7ff0ac96302a 0x55b0ba8844a9 0x55b0bb39b016 0x55b0ba88e742 0x55b0ba88e967 0x55b0ba88ebbf 0x55b0bbf8c47f 0x55b0bbf8c9cb 0x55b0bbf88fbd 0x55b0bbf8db97 0x55b0bbfa778b 0x55b0bbfe0b03 0x55b0bc06b36b 0x55b0bc06b8ad 0x55b0bc06bb6c 0x55b0bbfef201 0x55b0bc061718 0x55b0bc02b6be 0x55b0bc02b79b 0x55b0bbfdcd6d 0x55b0bb37fedf 0x55b0bb37875c 0x55b0bb26c590 0x55b0ba87050e 0x55b0ba88ff66 0x7ff0ac94c830 0x55b0ba8f5a49
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"55B0BA023000","o":"1663DF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55B0BA023000","o":"1663009"},{"b":"55B0BA023000","o":"16634ED"},{"b":"7FF0ACCF5000","o":"11390"},{"b":"7FF0AC92C000","o":"35428","s":"gsignal"},{"b":"7FF0AC92C000","o":"3702A","s":"abort"},{"b":"55B0BA023000","o":"8614A9","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55B0BA023000","o":"1378016"},{"b":"55B0BA023000","o":"86B742","s":"__wt_eventv"},{"b":"55B0BA023000","o":"86B967","s":"__wt_err"},{"b":"55B0BA023000","o":"86BBBF","s":"__wt_panic"},{"b":"55B0BA023000","o":"1F6947F","s":"__wt_block_extlist_read"},{"b":"55B0BA023000","o":"1F699CB","s":"__wt_block_extlist_read_avail"},{"b":"55B0BA023000","o":"1F65FBD","s":"__wt_block_checkpoint_load"},{"b":"55B0BA023000","o":"1F6AB97"},{"b":"55B0BA023000","o":"1F8478B","s":"__wt_btree_open"},{"b":"55B0BA023000","o":"1FBDB03","s":"__wt_conn_btree_open"},{"b":"55B0BA023000","o":"204836B","s":"__wt_session_get_btree"},{"b":"55B0BA023000","o":"20488AD","s":"__wt_session_get_btree"},{"b":"55B0BA023000","o":"2048B6C","s":"__wt_session_get_btree_ckpt"},{"b":"55B0BA023000","o":"1FCC201","s":"__wt_curfile_open"},{"b":"55B0BA023000","o":"203E718"},{"b":"55B0BA023000","o":"20086BE","s":"__wt_metadata_cursor_open"},{"b":"55B0BA023000","o":"200879B","s":"__wt_metadata_cursor"},{"b":"55B0BA023000","o":"1FB9D6D","s":"wiredtiger_open"},{"b":"55B0BA023000","o":"135CEDF","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55B0BA023000","o":"135575C"},{"b":"55B0BA023000","o":"1249590","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55B0BA023000","o":"84D50E"},{"b":"55B0BA023000","o":"86CF66","s":"main"},{"b":"7FF0AC92C000","o":"20830","s":"__libc_start_main"},{"b":"55B0BA023000","o":"8D2A49","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.6", "gitVersion" : "c55eb86ef46ee7aede3b1e2a5d184a7df4bfb5b5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "4.8.0-36-generic", "version" : "#36~16.04.1-Ubuntu SMP Sun Feb 5 09:39:57 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55B0BA023000", "elfType" : 3, "buildId" : "EAF1102647F381CAB3D8EC0A23EB4B023E96A5EC" }, { "b" : "7FFC45F76000", "elfType" : 3, "buildId" : "FE95293FF090A087E09855AADBE77230311AFFB1" }, { "b" : "7FF0AF7EB000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7FF0AF5A1000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DB5180B568097E2A4690A5B40D36BD134C893FEE" }, { "b" : "7FF0AF332000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "DED90CB11A7648E334E068A2585DB44E8CC07686" }, { "b" : "7FF0AEEB6000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "8FE272AED3883FEB05AC784E50D9D8F0FAC4E899" }, { "b" : "7FF0AECA7000", "path" : "/usr/lib/x86_64-linux-gnu/libsensors.so.4", "elfType" : 3, "buildId" : "1A855916EA4E082827D0599C11C0E82D98284118" }, { "b" : "7FF0AEA9A000", "path" : "/lib/x86_64-linux-gnu/libpci.so.3", "elfType" : 3, "buildId" : "3F90C8FDF5FC50256DD803E6F5EBED276A8193A6" }, { "b" : "7FF0AE896000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "AFDCB9B3774F5AFE6EB3EC9A76A8BA159A17D5B6" }, { "b" : "7FF0AE62D000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "901744249A5926FD226B85CBE594271ADFB80E52" }, { "b" : "7FF0AE423000", "path" : "/lib/x86_64-linux-gnu/libwrap.so.0", "elfType" : 3, "buildId" : "54FCBC5B0F994A13A9B0EAD46F23E7DA7F7FE75B" }, { "b" : "7FF0AE146000", "path" : "/usr/lib/x86_64-linux-gnu/libnetsnmp.so.30", "elfType" : 3, "buildId" : "0C029B406758FAFD5EC0E9B2C1F81D8D5FFC9D26" }, { "b" : "7FF0ADD02000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "E6D4D2E4A048992CD5501E5985094E6CEC6C5D4F" }, { "b" : "7FF0ADAB1000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "ADF6F41F223D42193165FA0C55871F02D915FB19" }, { "b" : "7FF0AD8A2000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "6B9F4061A1D44813A54DA4DBB0088F529D8D78EA" }, { "b" : "7FF0AD599000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4DBE0469DACBA00586428C9C7F4BBCEC8B6B1489" }, { "b" : "7FF0AD330000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "7F514146540382F59AD705BA8C913A75204C6858" }, { "b" : "7FF0AD128000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A656E8087D6226AE46A173477B8B8E0767D559D9" }, { "b" : "7FF0ACF12000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FF0ACCF5000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "4D7F52F335DC9665C2DCF308CE6514A6AE86DEDE" }, { "b" : "7FF0AC92C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "60131540DADC6796CAB33388349E6E4E68692053" }, { "b" : "7FF0AFA06000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "EA8652B03788783033C3AF093DFB71509C1378F8" }, { "b" : "7FF0AC65A000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "16E3DBC6D048145939BB43BBFD7954D27421B00F" }, { "b" : "7FF0AC42B000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "AEB4C08FC47F86C475E9D3996DFE5E9B403ACEBF" }, { "b" : "7FF0AC227000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7FF0AC01C000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "DF3219B89E86920E901BAC4A80AA60F2B6134588" }, { "b" : "7FF0ABDE9000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, { "b" : "7FF0ABBCD000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B15D78D75542A4CAEC1DB7F7FB7723817E58D536" }, { "b" : "7FF0AB9B3000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "340B7B463F981B8A0FB3451751F881DF1B0C2F74" }, { "b" : "7FF0AB798000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "1C39D66F4A377C4E71DA0BF23E7221B45DEF9E7F" }, { "b" : "7FF0AFBEA000", "path" : "/lib/x86_64-linux-gnu/libudev.so.1", "elfType" : 3, "buildId" : "677E00F6578E7FD54267F859E744E43667001142" }, { "b" : "7FF0AB3BD000", "path" : "/usr/lib/x86_64-linux-gnu/libperl.so.5.22", "elfType" : 3, "buildId" : "9F15B66FC5A0C22FB46B2B2E91F3CE705857A05E" }, { "b" : "7FF0AB1A4000", "path" : "/lib/x86_64-linux-gnu/libnsl.so.1", "elfType" : 3, "buildId" : "A393386743793EDF5A6DE8E5BF11C315E0ADD6FB" }, { "b" : "7FF0AAF63000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "40315538B8F467FA9379C26806C2C2E5F75DF62A" }, { "b" : "7FF0AAC33000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "B081530BA0EE8C7628C535957E776208328C3AD8" }, { "b" : "7FF0AAA2F000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FF0AA7FC000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FF0AA5C6000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FF0AA346000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FF0AA10E000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "C60ECF42664BABCC4F6D735D70051131C9FD28C9" }, { "b" : "7FF0A9F05000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "D72147C9E81DC821D2C5CA3A93757EE00DAC67EA" }, { "b" : "7FF0A9C7B000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "889221E5B7DFAA3D74CD8D55DAEF89DD962DAEB7" }, { "b" : "7FF0A99D9000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "B7A0E46FB5107A9566AB0D448C865DB7BB1390BF" }, { "b" : "7FF0A97A6000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "54027CA597C9AFC9D32A3D677857E5C8512C9FBF" }, { "b" : "7FF0A9590000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "AE98B3C2FEC7AEDB3064E03BD8C1883FD12898D1" }, { "b" : "7FF0A932C000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "A0E2D03FF5CF65937F4425D4EFD4D655243809EB" }, { "b" : "7FF0A9119000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "F5B37E248C55FD582ECD9B0240F96E8458F8E6F0" }, { "b" : "7FF0A8EF0000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "FEA2A08A986C1C954F7DED5A27616F0B4B18C6A2" }, { "b" : "7FF0A8CE1000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "E5BC16E04FC04459D5AB8F045B5F7C7838B86027" }, { "b" : "7FF0A8A96000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "6B6EF07C8D0FA1CE7CCEE58043BDB6C347F25544" }, { "b" : "7FF0A87C1000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7FF0A85B9000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55b0bb686df1]
       mongod(+0x1663009) [0x55b0bb686009]
       mongod(+0x16634ED) [0x55b0bb6864ed]
       libpthread.so.0(+0x11390) [0x7ff0acd06390]
       libc.so.6(gsignal+0x38) [0x7ff0ac961428]
       libc.so.6(abort+0x16A) [0x7ff0ac96302a]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55b0ba8844a9]
       mongod(+0x1378016) [0x55b0bb39b016]
       mongod(__wt_eventv+0x3D7) [0x55b0ba88e742]
       mongod(__wt_err+0x9D) [0x55b0ba88e967]
       mongod(__wt_panic+0x24) [0x55b0ba88ebbf]
       mongod(__wt_block_extlist_read+0x8F) [0x55b0bbf8c47f]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x55b0bbf8c9cb]
       mongod(__wt_block_checkpoint_load+0x26D) [0x55b0bbf88fbd]
       mongod(+0x1F6AB97) [0x55b0bbf8db97]
       mongod(__wt_btree_open+0xB3B) [0x55b0bbfa778b]
       mongod(__wt_conn_btree_open+0x163) [0x55b0bbfe0b03]
       mongod(__wt_session_get_btree+0xFB) [0x55b0bc06b36b]
       mongod(__wt_session_get_btree+0x63D) [0x55b0bc06b8ad]
       mongod(__wt_session_get_btree_ckpt+0x14C) [0x55b0bc06bb6c]
       mongod(__wt_curfile_open+0x161) [0x55b0bbfef201]
       mongod(+0x203E718) [0x55b0bc061718]
       mongod(__wt_metadata_cursor_open+0x6E) [0x55b0bc02b6be]
       mongod(__wt_metadata_cursor+0x4B) [0x55b0bc02b79b]
       mongod(wiredtiger_open+0x164D) [0x55b0bbfdcd6d]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x70F) [0x55b0bb37fedf]
       mongod(+0x135575C) [0x55b0bb37875c]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6B0) [0x55b0bb26c590]
       mongod(+0x84D50E) [0x55b0ba87050e]
       mongod(main+0x966) [0x55b0ba88ff66]
       libc.so.6(__libc_start_main+0xF0) [0x7ff0ac94c830]
       mongod(_start+0x29) [0x55b0ba8f5a49]
      -----  END BACKTRACE  -----
      
      

        1. dbpath_filelist.txt
          8 kB
        2. mongod_start.log
          12 kB
        3. mongod.log
          14 kB
        4. repair_attempt.tar.gz
          14 kB
        5. sizeStorer.wt
          44 kB
        6. WiredTiger
          0.0 kB
        7. WiredTiger.turtle
          1.0 kB
        8. WiredTiger.wt
          216 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            tsuyu Tsuyufumi WATANABE
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: