Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-49412

file:WiredTiger.wt read checksum error, mongodb won't start

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Done
    • Icon: Critical - P2 Critical - P2
    • None
    • None
    • None
    • None

    Description

      Hi, unfortuneatly I am having issues getting a mongodb started back up after a system failure.
      I have tried the mongod --repair, as well as using the wt tool to check, however they both report an invalid checksuom on the WiredTiger.wt file.
      I was hoping you ad a better way of attempting recovery on these?

      Thank you, logs below:

      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] MongoDB starting : pid=29 port=27017 dbpath=/var/lib/mongodb/data 64-bit host=mongodb-1-wnfnn
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] db version v3.4.9
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] allocator: tcmalloc
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] modules: none
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] build environment:
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-07-09T21:16:03.399+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:

      { bindIp: "localhost", port: 27017 }

      , replication: { oplogSizeMB: 64 }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb/data", wiredTiger: { engineConfig:

      { cacheSizeGB: 1.0 }

      } }, systemLog: { quiet: true } }
      MongoDB shell version v3.4.9
      2020-07-09T21:16:03.409+0000 I - [initandlisten] Detected data files in /var/lib/mongodb/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2020-07-09T21:16:03.411+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1024M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      connecting to: mongodb://127.0.0.1:27017/localhost
      2020-07-09T21:16:03.458+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
      2020-07-09T21:16:03.458+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
      connect@src/mongo/shell/mongo.js:237:13
      @(connect):1:6
      2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (0) [1594329363:465773][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 57344: block header checksum of 979725410 doesn't match expected checksum of 793883532
      2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (0) [1594329363:465887][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2020-07-09T21:16:03.465+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1594329363:465909][29:0x7f47e1b69e80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2020-07-09T21:16:03.465+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      2020-07-09T21:16:03.465+0000 I - [initandlisten]

      ***aborting after fassert() failure

      exception: connect failed
      2020-07-09T21:16:03.470+0000 F - [initandlisten] Got signal: 6 (Aborted).

      0x56354b05735a 0x56354b056699 0x56354b056b66 0x7f47de49e6d0 0x7f47de0f8277 0x7f47de0f9968 0x56354a2b745b 0x56354ad5cc26 0x56354a2c077a 0x56354a2c0991 0x56354a2c0bef 0x56354b84887f 0x56354b848ddb 0x56354b845435 0x56354b849fa7 0x56354b863a35 0x56354b89d3f3 0x56354b928747 0x56354b928c96 0x56354b928f5c 0x56354b8abb31 0x56354b91e9f8 0x56354b8e82be 0x56354b8e83ab 0x56354b899777 0x56354ad42031 0x56354ad39fc6 0x56354ac20517 0x56354a29d984 0x56354a2c1e44 0x7f47de0e4445 0x56354a31f615
      ----- BEGIN BACKTRACE -----

      {"backtrace":[\{"b":"56354A0B4000","o":"FA335A"}

      ,{"b":"56354A0B4000","o":"FA2699"},{"b":"56354A0B4000","o":"FA2B66"},{"b":"7F47DE48F000","o":"F6D0"},{"b":"7F47DE0C2000","o":"36277","s":"gsignal"},{"b":"7F47DE0C2000","o":"37968","s":"abort"},{"b":"56354A0B4000","o":"20345B"},{"b":"56354A0B4000","o":"CA8C26"},{"b":"56354A0B4000","o":"20C77A"},{"b":"56354A0B4000","o":"20C991","s":"_wt_err"},{"b":"56354A0B4000","o":"20CBEF","s":"wt_panic"},{"b":"56354A0B4000","o":"179487F"},{"b":"56354A0B4000","o":"1794DDB"},{"b":"56354A0B4000","o":"1791435"},{"b":"56354A0B4000","o":"1795FA7"},{"b":"56354A0B4000","o":"17AFA35"},{"b":"56354A0B4000","o":"17E93F3"},{"b":"56354A0B4000","o":"1874747"},{"b":"56354A0B4000","o":"1874C96"},{"b":"56354A0B4000","o":"1874F5C"},{"b":"56354A0B4000","o":"17F7B31"},{"b":"56354A0B4000","o":"186A9F8"},{"b":"56354A0B4000","o":"18342BE"},{"b":"56354A0B4000","o":"18343AB"},{"b":"56354A0B4000","o":"17E5777","s":"wiredtiger_open"},{"b":"56354A0B4000","o":"C8E031"},{"b":"56354A0B4000","o":"C85FC6"},{"b":"56354A0B4000","o":"B6C517"},{"b":"56354A0B4000","o":"1E9984"},{"b":"56354A0B4000","o":"20DE44"},{"b":"7F47DE0C2000","o":"22445","s":"_libc_start_main"},{"b":"56354A0B4000","o":"26B615"}],"processInfo":{ "mongodbVersion" : "3.4.9", "gitVersion" : "876ebee8c7dd0e2d992f36a848ff4dc50ee6603e", "compiledModules" : [], "uname" :

      { "sysname" : "Linux", "release" : "3.10.0-1127.13.1.el7.x86_64", "version" : "#1 SMP Tue Jun 23 15:46:38 UTC 2020", "machine" : "x86_64" }

      , "somap" : [ { "b" : "56354A0B4000", "elfType" : 3, "buildId" : "28974745B6B8764D64F97F6BF19D04AC5EDD5B65" }, { "b" : "7FFE6449A000", "elfType" : 3, "buildId" : "B1962B2DF9C13102C04B311EBA759764C1B8E9D8" }, { "b" : "7F47E1749000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F47E14C0000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libyaml-cpp.so.rh-mongodb34-0.5", "elfType" : 3, "buildId" : "FE3E65529D3D5623ECF32A331FFBA7ACD02D40A6" }, { "b" : "7F47E12BA000", "path" : "/lib64/libsnappy.so.1", "elfType" : 3, "buildId" : "3CEB901120465B031FA92FA449A9AD981CEC1659" }, { "b" : "7F47E1048000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libtcmalloc.so.rh-mongodb34-4", "elfType" : 3, "buildId" : "9B8AE99A244389054FED1DC74C0F7B79CD6920CE" }, { "b" : "7F47E0DE6000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F47E0BDD000", "path" : "/lib64/libpcrecpp.so.0", "elfType" : 3, "buildId" : "8BBAA8A5638DCB0C4B523FCA4B54613AE543BDF1" }, { "b" : "7F47E096A000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_program_options.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "51F7B5995F6FD852D84CE889AED4DF5807C588FD" }, { "b" : "7F47E0753000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_filesystem.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "BC537D140DFB358B75B19A985EDE485EB9D122EE" }, { "b" : "7F47E0530000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_thread.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "66EB1D274A8B36597AE3AEDB3A86730FD37F5317" }, { "b" : "7F47E032C000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_system.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "5BCE91B1A7B7AAA44098CF23DA1B795F62F2B1D4" }, { "b" : "7F47E0114000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_iostreams.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "5FF6A64EF26564056ECEB54D3BD7923CEDB27112" }, { "b" : "7F47DFF0C000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_chrono.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "413AD9DB5A13FFC6E5252AD320DC2C84D17753DF" }, { "b" : "7F47DFBFC000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libboost_regex.so.rh-mongodb34-1.60.0", "elfType" : 3, "buildId" : "C581BE7B9B2C0A59BC05D34F6922A66C5D89D475" }, { "b" : "7F47DF9A9000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libstemmer.so.rh-mongodb34-0", "elfType" : 3, "buildId" : "EAAFF5B4F644622B2E88F23A70098C539A7DA9E8" }, { "b" : "7F47DF737000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7F47DF2D6000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7F47DF0CE000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7F47DEECA000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CDB5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7F47DEBC3000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "9589AE0FDA6AEB1183EBA1C62A328F933E7817FD" }, { "b" : "7F47DE8C1000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7F47DE6AB000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "531AA1391EA4E1489D5EF11AA5DC2FFD9E2BDFEE" }, { "b" : "7F47DE48F000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7F47DE0C2000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7F47DDEA8000", "path" : "/opt/rh/rh-mongodb34/root/usr/lib64/libunwind.so.rh-mongodb34-8", "elfType" : 3, "buildId" : "CE5996C64EA0376E71D6F37FCEC1FB95F860972F" }, { "b" : "7F47E195F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7F47DDC98000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "0C85C0386F0CF41EA39969CF7F58A558D1AD3235" }, { "b" : "7F47DC6C4000", "path" : "/lib64/libicudata.so.50", "elfType" : 3, "buildId" : "04813117A9FB30F2DB02B8C546C4FC38BF4890A0" }, { "b" : "7F47DC2C5000", "path" : "/lib64/libicui18n.so.50", "elfType" : 3, "buildId" : "7CAA9714B33AA18B16FAFDC88FA43185238C6C86" }, { "b" : "7F47DBF4C000", "path" : "/lib64/libicuuc.so.50", "elfType" : 3, "buildId" : "E9EA46B17EEE2AE15094650C00B891C094CB802D" }, { "b" : "7F47DBCFF000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "16FE0DC6CEFC5F444BC876516D02EFE9CC2D432F" }, { "b" : "7F47DBA17000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D1CD1B94855A85FBC735C745DB39BC096F7D8CC3" }, { "b" : "7F47DB813000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2C7EF64EF0C5AF8BCFA8F9E628E5605A7D8C52D3" }, { "b" : "7F47DB5E0000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A20F715C514B3EA873F4CC77D585A50CB670E266" }, { "b" : "7F47DB3D2000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "9F82B06CE44724A8ACE827A2C95A9A76619EA314" }, { "b" : "7F47DB1CE000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F47DAFB5000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7F47DAD8E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "1B0C0DC4F9054C5922082921C1447739F27AC8F4" } ] }}
      mongod(+0xFA335A) [0x56354b05735a]
      mongod(+0xFA2699) [0x56354b056699]
      mongod(+0xFA2B66) [0x56354b056b66]
      libpthread.so.0(+0xF6D0) [0x7f47de49e6d0]
      libc.so.6(gsignal+0x37) [0x7f47de0f8277]
      libc.so.6(abort+0x148) [0x7f47de0f9968]
      mongod(+0x20345B) [0x56354a2b745b]
      mongod(+0xCA8C26) [0x56354ad5cc26]
      mongod(+0x20C77A) [0x56354a2c077a]
      mongod(__wt_err+0xA0) [0x56354a2c0991]
      mongod(__wt_panic+0x24) [0x56354a2c0bef]
      mongod(+0x179487F) [0x56354b84887f]
      mongod(+0x1794DDB) [0x56354b848ddb]
      mongod(+0x1791435) [0x56354b845435]
      mongod(+0x1795FA7) [0x56354b849fa7]
      mongod(+0x17AFA35) [0x56354b863a35]
      mongod(+0x17E93F3) [0x56354b89d3f3]
      mongod(+0x1874747) [0x56354b928747]
      mongod(+0x1874C96) [0x56354b928c96]
      mongod(+0x1874F5C) [0x56354b928f5c]
      mongod(+0x17F7B31) [0x56354b8abb31]
      mongod(+0x186A9F8) [0x56354b91e9f8]
      mongod(+0x18342BE) [0x56354b8e82be]
      mongod(+0x18343AB) [0x56354b8e83ab]
      mongod(wiredtiger_open+0x1767) [0x56354b899777]
      mongod(+0xC8E031) [0x56354ad42031]
      mongod(+0xC85FC6) [0x56354ad39fc6]
      mongod(+0xB6C517) [0x56354ac20517]
      mongod(+0x1E9984) [0x56354a29d984]
      mongod(+0x20DE44) [0x56354a2c1e44]
      libc.so.6(__libc_start_main+0xF5) [0x7f47de0e4445]
      mongod(+0x26B615) [0x56354a31f615]
      ----- END BACKTRACE -----

      Attachments

        1. WiredTiger.turtle
          1.0 kB
        2. WiredTiger.wt
          400 kB

        Activity

          People

            dmitry.agranat@mongodb.com Dmitry Agranat
            mpeterson@rglholdings.com Michael Peterson
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: