Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-40458

Unexpected server shutdown and corrupted data

    XMLWordPrintable

Details

    • Question
    • Status: Closed
    • Major - P3
    • Resolution: Done
    • 3.4.20
    • None
    • None

    Description

      Hello!
      We had unexpected server shutdown and it made our database corrupt, we wanted to be sure if it is possible to get back at least some data out of corruption?
      We had no luck with  --repair and other commands this far.
      We are using mongodb v.3.4.2, here is our log output, i have replaced db path and my hostname with ***** -

      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] MongoDB starting : pid=931 port=27017 dbpath=/*****/ 64-bit host=*****
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] db version v3.4.2
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] allocator: tcmalloc
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] modules: none
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] build environment:
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] distmod: ubuntu1604
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] distarch: x86_64
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] target_arch: x86_64
      2019-04-03T14:24:25.604+0300 I CONTROL [initandlisten] options: { net: { port: 27017 }, replication: { oplogSizeMB: 128, replSet: "rs0" }, storage: { dbPath: "/*****/", mmapv1: { smallFiles: true } } }
      2019-04-03T14:24:25.632+0300 I - [initandlisten] Detected data files in /*****/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten]
      2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2019-04-03T14:24:25.632+0300 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
      2019-04-03T14:24:25.633+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7453M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (0) [1554290665:703634][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 24576: block header checksum of 2396726276 doesn't match expected checksum of 1875689742
      2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (0) [1554290665:703692][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2019-04-03T14:24:25.703+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1554290665:703699][931:0x7fe3ea5a9cc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2019-04-03T14:24:25.703+0300 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      2019-04-03T14:24:25.703+0300 I - [initandlisten]
      ***aborting after fassert() failure
       
      2019-04-03T14:24:25.724+0300 F - [initandlisten] Got signal: 6 (Aborted).
      0x5652eae43b81 0x5652eae42c79 0x5652eae4315d 0x7fe3e91cd390 0x7fe3e8e28428 0x7fe3e8e2a02a 0x5652ea0d2293 0x5652eab536c6 0x5652ea0dc505 0x5652ea0dc5f9 0x5652ea0dc851 0x5652eb7442bf 0x5652eb74480b 0x5652eb740f1d 0x5652eb7459e7 0x5652eb763126 0x5652eb79988b 0x5652eb82668d 0x5652eb826d88 0x5652eb82729c 0x5652eb7a9681 0x5652eb81c0c0 0x5652eb7e653e 0x5652eb7e662b 0x5652eb795d13 0x5652eab36cf6 0x5652eab2fb8c 0x5652eaa1f3d0 0x5652ea0bdb7e 0x5652ea0ddc3f 0x7fe3e8e13830 0x5652ea13b979
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"5652E98C1000","o":"1582B81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5652E98C1000","o":"1581C79"},{"b":"5652E98C1000","o":"158215D"},{"b":"7FE3E91BC000","o":"11390"},{"b":"7FE3E8DF3000","o":"35428","s":"gsignal"},{"b":"7FE3E8DF3000","o":"3702A","s":"abort"},{"b":"5652E98C1000","o":"811293","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"5652E98C1000","o":"12926C6"},{"b":"5652E98C1000","o":"81B505"},{"b":"5652E98C1000","o":"81B5F9","s":"__wt_err"},{"b":"5652E98C1000","o":"81B851","s":"__wt_panic"},{"b":"5652E98C1000","o":"1E832BF"},{"b":"5652E98C1000","o":"1E8380B"},{"b":"5652E98C1000","o":"1E7FF1D"},{"b":"5652E98C1000","o":"1E849E7"},{"b":"5652E98C1000","o":"1EA2126"},{"b":"5652E98C1000","o":"1ED888B"},{"b":"5652E98C1000","o":"1F6568D"},{"b":"5652E98C1000","o":"1F65D88"},{"b":"5652E98C1000","o":"1F6629C"},{"b":"5652E98C1000","o":"1EE8681"},{"b":"5652E98C1000","o":"1F5B0C0"},{"b":"5652E98C1000","o":"1F2553E"},{"b":"5652E98C1000","o":"1F2562B"},{"b":"5652E98C1000","o":"1ED4D13","s":"wiredtiger_open"},{"b":"5652E98C1000","o":"1275CF6","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"5652E98C1000","o":"126EB8C"},{"b":"5652E98C1000","o":"115E3D0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"5652E98C1000","o":"7FCB7E"},{"b":"5652E98C1000","o":"81CC3F","s":"main"},{"b":"7FE3E8DF3000","o":"20830","s":"__libc_start_main"},{"b":"5652E98C1000","o":"87A979","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-53-generic", "version" : "#74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "5652E98C1000", "elfType" : 3, "buildId" : "5369A3377E676C284ECBB17624E909071B0846FC" }, { "b" : "7FFF7FBE5000", "elfType" : 3, "buildId" : "E3616EB2EA9FD27AC9DF53212C1C57A8A091F03D" }, { "b" : "7FE3EA148000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "99E596AA738A74440C0300BFF1086C8C15455279" }, { "b" : "7FE3E9D04000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3F3A35F9ABBE347B08E59845A1FC8EB5AE226797" }, { "b" : "7FE3E9AFC000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "0DBB8C21FC5D977098CA718BA2BFD6C4C21172E9" }, { "b" : "7FE3E98F8000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C0C5B7F18348654040534B050B110D32A19EA38D" }, { "b" : "7FE3E95EF000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "05451CB4D66C321691F64F253880B7CE5B8812A6" }, { "b" : "7FE3E93D9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FE3E91BC000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "84538E3C6CFCD5D4E3C0D2B6C3373F802915A498" }, { "b" : "7FE3E8DF3000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CBFA941A8EB7A11E4F90E81B66FCD5A820995D7C" }, { "b" : "7FE3EA3B1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A7D5A820B802049276B1FC26C8E845A3E194EB6B" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5652eae43b81]
       mongod(+0x1581C79) [0x5652eae42c79]
       mongod(+0x158215D) [0x5652eae4315d]
       libpthread.so.0(+0x11390) [0x7fe3e91cd390]
       libc.so.6(gsignal+0x38) [0x7fe3e8e28428]
       libc.so.6(abort+0x16A) [0x7fe3e8e2a02a]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5652ea0d2293]
       mongod(+0x12926C6) [0x5652eab536c6]
       mongod(+0x81B505) [0x5652ea0dc505]
       mongod(__wt_err+0x9D) [0x5652ea0dc5f9]
       mongod(__wt_panic+0x24) [0x5652ea0dc851]
       mongod(+0x1E832BF) [0x5652eb7442bf]
       mongod(+0x1E8380B) [0x5652eb74480b]
       mongod(+0x1E7FF1D) [0x5652eb740f1d]
       mongod(+0x1E849E7) [0x5652eb7459e7]
       mongod(+0x1EA2126) [0x5652eb763126]
       mongod(+0x1ED888B) [0x5652eb79988b]
       mongod(+0x1F6568D) [0x5652eb82668d]
       mongod(+0x1F65D88) [0x5652eb826d88]
       mongod(+0x1F6629C) [0x5652eb82729c]
       mongod(+0x1EE8681) [0x5652eb7a9681]
       mongod(+0x1F5B0C0) [0x5652eb81c0c0]
       mongod(+0x1F2553E) [0x5652eb7e653e]
       mongod(+0x1F2562B) [0x5652eb7e662b]
       mongod(wiredtiger_open+0x1713) [0x5652eb795d13]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xA76) [0x5652eab36cf6]
       mongod(+0x126EB8C) [0x5652eab2fb8c]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x5652eaa1f3d0]
       mongod(+0x7FCB7E) [0x5652ea0bdb7e]
       mongod(main+0x95F) [0x5652ea0ddc3f]
       libc.so.6(__libc_start_main+0xF0) [0x7fe3e8e13830]
       mongod(_start+0x29) [0x5652ea13b979]
      ----- END BACKTRACE -----
      Aborted
      

      I have also attached .turtle and .wt WiredTiger files.

      Hope to hear from You soon

      Attachments

        1. collection-2--9014212594577530165.wt
          68.74 MB
        2. SERVER-40458_repair_attempt.tgz
          15 kB
        3. WiredTiger.turtle
          1.0 kB
        4. WiredTiger.wt
          272 kB

        Activity

          People

            eric.sedor@mongodb.com Eric Sedor
            raivis.ginters Raivis Ginters
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: