Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44417

Fatal Assertion WiredTiger error

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: WiredTiger
    • Labels:
      None
    • ALL

      Hi Team,
      We are facing this problem after a power failure on our server. I try to run --repair but it couldn't solve the problem. Its important for us to have back the data like everyone else, its around 1.1TB. By the way we had a replica set for backup but in the power failure other servers hdds had gone so this is our last chance.
      Here is my log and wt files

      2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] MongoDB starting : pid=3182 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb2
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] db version v3.6.1
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] allocator: tcmalloc
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] modules: none
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] build environment:
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] distmod: ubuntu1604
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] distarch: x86_64
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] target_arch: x86_64
       2019-11-05T13:05:32.320+0300 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net:
      
      { bindIp: "127.0.0.1,192.168.2.11", port: 27017 }
      
      , processManagement:
      
      { timeZoneInfo: "/usr/share/zoneinfo" }
      
      , replication:
      
      { replSetName: "isea" }
      
      , storage: { dbPath: "/var/lib/mongodb", journal:
      
      { enabled: true }
      
      }, systemLog:
      
      { destination: "file", logAppend: false, path: "/var/log/mongodb/mongod.log" }
      
      }
       2019-11-05T13:05:32.322+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
       2019-11-05T13:05:32.323+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=31702M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
       2019-11-05T13:05:32.350+0300 E STORAGE [initandlisten] WiredTiger error (0) [1572948332:350144][3182:0x7f0608d009c0], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: read checksum error for 4096B block at offset 12288: block header checksum of 2731310484 doesn't match expected checksum of 1144327562
       2019-11-05T13:05:32.350+0300 E STORAGE [initandlisten] WiredTiger error (0) [1572948332:350193][3182:0x7f0608d009c0], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227)
       2019-11-05T13:05:32.350+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1572948332:350206][3182:0x7f0608d009c0], [file:WiredTiger.wt|file:///WiredTiger.wt], connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
       2019-11-05T13:05:32.350+0300 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
       2019-11-05T13:05:32.350+0300 F - [initandlisten]
      
      ***aborting after fassert() failure
      
      2019-11-05T13:05:32.391+0300 F - [initandlisten] Got signal: 6 (Aborted).
      
      0x5619b54751f1 0x5619b5474409 0x5619b54748ed 0x7f06076f8390 0x7f0607352428 0x7f060735402a 0x5619b3c0deff 0x5619b3cda43e 0x5619b3bab277 0x5619b3bab49c 0x5619b3bab7c5 0x5619b3def41f 0x5619b3def96b 0x5619b3debf25 0x5619b3dd5ec7 0x5619b3d588c7 0x5619b3cea256 0x5619b3d36c7b 0x5619b3d3719d 0x5619b3d3745c 0x5619b3d9fa72 0x5619b3d2da08 0x5619b3d03afe 0x5619b3d03bdb 0x5619b3ce8dd3 0x5619b3cbe7f5 0x5619b3ca2f28 0x5619b3e90170 0x5619b3ba7080 0x5619b3c81fdb 0x5619b3c0fbc9 0x7f060733d830 0x5619b3c71a19
       ----- BEGIN BACKTRACE -----
       {"backtrace":[
      
      {"b":"5619B3283000","o":"21F21F1","s":"_ZN5mongo15printStackTraceERSo"}
      
      ,
      
      {"b":"5619B3283000","o":"21F1409"}
      
      ,
      
      {"b":"5619B3283000","o":"21F18ED"}
      
      ,
      
      {"b":"7F06076E7000","o":"11390"}
      
      ,
      
      {"b":"7F060731D000","o":"35428","s":"gsignal"}
      
      ,
      
      {"b":"7F060731D000","o":"3702A","s":"abort"}
      
      ,
      
      {"b":"5619B3283000","o":"98AEFF","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"}
      
      ,
      
      {"b":"5619B3283000","o":"A5743E"}
      
      ,
      
      {"b":"5619B3283000","o":"928277","s":"__wt_eventv"}
      
      ,
      
      {"b":"5619B3283000","o":"92849C","s":"__wt_err"}
      
      ,
      
      {"b":"5619B3283000","o":"9287C5","s":"__wt_panic"}
      
      ,
      
      {"b":"5619B3283000","o":"B6C41F","s":"__wt_block_extlist_read"}
      
      ,
      
      {"b":"5619B3283000","o":"B6C96B","s":"__wt_block_extlist_read_avail"}
      
      ,
      
      {"b":"5619B3283000","o":"B68F25","s":"__wt_block_checkpoint_load"}
      
      ,
      
      {"b":"5619B3283000","o":"B52EC7"}
      
      ,
      
      {"b":"5619B3283000","o":"AD58C7","s":"__wt_btree_open"}
      
      ,
      
      {"b":"5619B3283000","o":"A67256","s":"__wt_conn_dhandle_open"}
      
      ,
      
      {"b":"5619B3283000","o":"AB3C7B","s":"__wt_session_get_dhandle"}
      
      ,
      
      {"b":"5619B3283000","o":"AB419D","s":"__wt_session_get_dhandle"}
      
      ,
      
      {"b":"5619B3283000","o":"AB445C","s":"__wt_session_get_btree_ckpt"}
      
      ,
      
      {"b":"5619B3283000","o":"B1CA72","s":"__wt_curfile_open"}
      
      ,
      
      {"b":"5619B3283000","o":"AAAA08"}
      
      ,
      
      {"b":"5619B3283000","o":"A80AFE","s":"__wt_metadata_cursor_open"}
      
      ,
      
      {"b":"5619B3283000","o":"A80BDB","s":"__wt_metadata_cursor"}
      
      ,
      
      {"b":"5619B3283000","o":"A65DD3","s":"wiredtiger_open"}
      
      ,
      
      {"b":"5619B3283000","o":"A3B7F5","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"}
      
      ,
      
      {"b":"5619B3283000","o":"A1FF28"}
      
      ,
      
      {"b":"5619B3283000","o":"C0D170","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"}
      
      ,
      
      {"b":"5619B3283000","o":"924080"}
      
      ,
      
      {"b":"5619B3283000","o":"9FEFDB","s":"_ZN5mongo11mongoDbMainEiPPcS1_"}
      
      ,
      
      {"b":"5619B3283000","o":"98CBC9","s":"main"}
      
      ,
      
      {"b":"7F060731D000","o":"20830","s":"__libc_start_main"}
      
      ,
      
      {"b":"5619B3283000","o":"9EEA19","s":"_start"}
      
      ],"processInfo":{ "mongodbVersion" : "3.6.1", "gitVersion" : "025d4f4fe61efd1fb6f0005be20cb45a004093d1", "compiledModules" : [], "uname" :
      
      { "sysname" : "Linux", "release" : "4.4.0-166-generic", "version" : "#195-Ubuntu SMP Tue Oct 1 09:35:25 UTC 2019", "machine" : "x86_64" }
      
      , "somap" : [
      
      { "b" : "5619B3283000", "elfType" : 3, "buildId" : "87C7D2729B4125CF6CBE2FD954E1BBD0441BBA53" }
      
      ,
      
      { "b" : "7FFC745C7000", "elfType" : 3, "buildId" : "183A8671B5C7AD4961111F66700B5B4195D08D82" }
      
      ,
      
      { "b" : "7F06088DD000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }
      
      ,
      
      { "b" : "7F0608674000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "FF69EA60EBE05F2DD689D2B26FC85A73E5FBC3A0" }
      
      ,
      
      { "b" : "7F060822F000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "15FFEB43278726B025F020862BF51302822A40EC" }
      
      ,
      
      { "b" : "7F0608027000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }
      
      ,
      
      { "b" : "7F0607E23000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }
      
      ,
      
      { "b" : "7F0607B1A000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }
      
      ,
      
      { "b" : "7F0607904000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }
      
      ,
      
      { "b" : "7F06076E7000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }
      
      ,
      
      { "b" : "7F060731D000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }
      
      ,
      
      { "b" : "7F0608AF8000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }
      
      ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5619b54751f1]
       mongod(+0x21F1409) [0x5619b5474409]
       mongod(+0x21F18ED) [0x5619b54748ed]
       libpthread.so.0(+0x11390) [0x7f06076f8390]
       libc.so.6(gsignal+0x38) [0x7f0607352428]
       libc.so.6(abort+0x16A) [0x7f060735402a]
       mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5619b3c0deff]
       mongod(+0xA5743E) [0x5619b3cda43e]
       mongod(__wt_eventv+0x3D7) [0x5619b3bab277]
       mongod(__wt_err+0x9D) [0x5619b3bab49c]
       mongod(__wt_panic+0x33) [0x5619b3bab7c5]
       mongod(__wt_block_extlist_read+0x8F) [0x5619b3def41f]
       mongod(__wt_block_extlist_read_avail+0x2B) [0x5619b3def96b]
       mongod(__wt_block_checkpoint_load+0x275) [0x5619b3debf25]
       mongod(+0xB52EC7) [0x5619b3dd5ec7]
       mongod(__wt_btree_open+0xBE7) [0x5619b3d588c7]
       mongod(__wt_conn_dhandle_open+0x276) [0x5619b3cea256]
       mongod(__wt_session_get_dhandle+0xFB) [0x5619b3d36c7b]
       mongod(__wt_session_get_dhandle+0x61D) [0x5619b3d3719d]
       mongod(__wt_session_get_btree_ckpt+0x14C) [0x5619b3d3745c]
       mongod(__wt_curfile_open+0x52) [0x5619b3d9fa72]
       mongod(+0xAAAA08) [0x5619b3d2da08]
       mongod(__wt_metadata_cursor_open+0x6E) [0x5619b3d03afe]
       mongod(__wt_metadata_cursor+0x4B) [0x5619b3d03bdb]
       mongod(wiredtiger_open+0x19C3) [0x5619b3ce8dd3]
       mongod(_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x845) [0x5619b3cbe7f5]
       mongod(+0xA1FF28) [0x5619b3ca2f28]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x650) [0x5619b3e90170]
       mongod(+0x924080) [0x5619b3ba7080]
       mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x86B) [0x5619b3c81fdb]
       mongod(main+0x9) [0x5619b3c0fbc9]
       libc.so.6(__libc_start_main+0xF0) [0x7f060733d830]
       mongod(_start+0x29) [0x5619b3c71a19]
       ----- END BACKTRACE -----
      

      We kindly request your help
      Thanks in advance.

        1. V4.0.12_repair_out.txt
          243 kB
        2. WiredTiger.turtle
          1 kB
        3. WiredTiger.wt
          4.69 MB

            Assignee:
            carl.champain@mongodb.com Carl Champain (Inactive)
            Reporter:
            anilatav@gmail.com ANIL ATAV
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: