Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-41906

file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.5
    • Component/s: None
    • Labels:
    • Environment:
      linux redhat7.3

      昨晚断电后,今天mongodb不能正常启动,尝试使用--repair修复,

      附件是config的错误和四组的问题文件,希望能帮助我修复这些问题文件,十分感谢,

      结果报错如下:

      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] MongoDB starting : pid=18883 port=27017 dbpath=/mongodbdata/config 64-bit host=sc-mongodb-02
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] db version v3.4.5
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] git version: 520b8f3092c48d934f0cd78ab5f40fe594f96863
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] allocator: tcmalloc
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] modules: none
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] build environment:
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] distmod: rhel62
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] distarch: x86_64
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] target_arch: x86_64
      2019-06-25T12:39:53.299+0800 I CONTROL [initandlisten] options: { repair: true, storage:

      { dbPath: "/mongodbdata/config" }

      , systemLog: { destination: "file", path: "/mongodbdata/logs/config.log" } }
      2019-06-25T12:39:53.322+0800 I - [initandlisten] Detected data files in /mongodbdata/config created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2019-06-25T12:39:53.322+0800 I STORAGE [initandlisten]
      2019-06-25T12:39:53.322+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2019-06-25T12:39:53.322+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
      2019-06-25T12:39:53.322+0800 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint.
      2019-06-25T12:39:53.322+0800 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=31510M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2019-06-25T12:39:53.328+0800 E STORAGE [initandlisten] WiredTiger error (0) [1561437593:328949][18883:0x7fc246fabe40], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 859664997 doesn't match expected checksum of 4092033674
      2019-06-25T12:39:53.328+0800 E STORAGE [initandlisten] WiredTiger error (0) [1561437593:328975][18883:0x7fc246fabe40], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
      2019-06-25T12:39:53.328+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1561437593:328984][18883:0x7fc246fabe40], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2019-06-25T12:39:53.328+0800 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      2019-06-25T12:39:53.329+0800 I - [initandlisten]

      ***aborting after fassert() failure

      2019-06-25T12:39:53.349+0800 F - [initandlisten] Got signal: 6 (Aborted).

      0x561793079d51 0x561793078f69 0x56179307944d 0x7fc245ba55e0 0x7fc2458081f7 0x7fc2458098e8 0x561792321fab 0x561792d8c636 0x56179232c2ba 0x56179232c4d6 0x56179232c72e 0x56179398100f 0x56179398155b 0x56179397db4d 0x561793982727 0x56179399c32b 0x5617939d5643 0x561793a5fdeb 0x561793a6032d 0x561793a605ec 0x5617939e3e11 0x561793a561b8 0x561793a201fe 0x561793a202db 0x5617939d1949 0x561792d70980 0x561792d696e5 0x561792c5c397 0x56179230e19c 0x56179232dadb 0x7fc2457f4c05 0x56179238c2b1
      ----- BEGIN BACKTRACE -----

      {"backtrace":[\{"b":"561791B08000","o":"1571D51","s":"_ZN5mongo15printStackTraceERSo"}

      ,{"b":"561791B08000","o":"1570F69"},{"b":"561791B08000","o":"157144D"},{"b":"7FC245B96000","o":"F5E0"},{"b":"7FC2457D3000","o":"351F7","s":"gsignal"},{"b":"7FC2457D3000","o":"368E8","s":"abort"},{"b":"561791B08000","o":"819FAB","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"561791B08000","o":"1284636"},{"b":"561791B08000","o":"8242BA","s":"wt_eventv"},{"b":"561791B08000","o":"8244D6","s":"wt_err"},{"b":"561791B08000","o":"82472E","s":"wt_panic"},{"b":"561791B08000","o":"1E7900F","s":"wt_block_extlist_read"},{"b":"561791B08000","o":"1E7955B","s":"wt_block_extlist_read_avail"},{"b":"561791B08000","o":"1E75B4D","s":"wt_block_checkpoint_load"},{"b":"561791B08000","o":"1E7A727"},{"b":"561791B08000","o":"1E9432B","s":"wt_btree_open"},{"b":"561791B08000","o":"1ECD643","s":"wt_conn_btree_open"},{"b":"561791B08000","o":"1F57DEB","s":"wt_session_get_btree"},{"b":"561791B08000","o":"1F5832D","s":"wt_session_get_btree"},{"b":"561791B08000","o":"1F585EC","s":"wt_session_get_btree_ckpt"},{"b":"561791B08000","o":"1EDBE11","s":"wt_curfile_open"},{"b":"561791B08000","o":"1F4E1B8"},{"b":"561791B08000","o":"1F181FE","s":"wt_metadata_cursor_open"},{"b":"561791B08000","o":"1F182DB","s":"wt_metadata_cursor"},{"b":"561791B08000","o":"1EC9949","s":"wiredtiger_open"},{"b":"561791B08000","o":"1268980","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"561791B08000","o":"12616E5"},{"b":"561791B08000","o":"1154397","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"561791B08000","o":"80619C"},{"b":"561791B08000","o":"825ADB","s":"main"},{"b":"7FC2457D3000","o":"21C05","s":"_libc_start_main"},{"b":"561791B08000","o":"8842B1"}],"processInfo":{ "mongodbVersion" : "3.4.5", "gitVersion" : "520b8f3092c48d934f0cd78ab5f40fe594f96863", "compiledModules" : [], "uname" :

      { "sysname" : "Linux", "release" : "4.1.12-61.1.18.el7uek.x86_64", "version" : "#2 SMP Fri Nov 4 15:48:30 PDT 2016", "machine" : "x86_64" }

      , "somap" : [ { "b" : "561791B08000", "elfType" : 3, "buildId" : "136B0D43602CC0EC1B0B2EDC5823DF0F16BFA594" }, { "b" : "7FFF59144000", "elfType" : 3, "buildId" : "AF290EF0E9960C89F2EA0B7AB86321368D658ED9" }, { "b" : "7FC246B37000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "1712F00C3A44AE54FBFD332C452A56476702A26E" }, { "b" : "7FC2466D6000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "6321431B47CBF126621AF97BD81303A1FC85C1B8" }, { "b" : "7FC2464CE000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "76C4765826961C60BE913F5549EBBC7BBE506899" }, { "b" : "7FC2462CA000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "EF023E0C16991E8957337C90B0E51025FE27C897" }, { "b" : "7FC245FC8000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6A344A5CE77A247399AA2F22F92C698F574A7134" }, { "b" : "7FC245DB2000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3E85E6D20D2CE9CDAD535084BEA56620BAAD687C" }, { "b" : "7FC245B96000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "6091139B61D901AA8425DBC16B7F51D455A416B4" }, { "b" : "7FC2457D3000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "BA7D7EBB462434829BD0E215446B0D3C2E47BEEC" }, { "b" : "7FC246DA9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A9980CF253C79740E69F70DCB8FEA7B8C2F641B5" }, { "b" : "7FC245586000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DC2687F4E7034B1175A0351CB17D9AFE14D3B2E2" }, { "b" : "7FC24529E000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "700752C8E20EBF87C748DDD6C847BFC49D8183D1" }, { "b" : "7FC24509A000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "1334477BC12948138DC4DDB4C2771A345B9F697D" }, { "b" : "7FC244E67000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "9F4CF016958F7E9CBEFC1BDEE328781D81BF72E9" }, { "b" : "7FC244C51000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "FE621E91052A9A77CC263E00A8A21C2BC0867E21" }, { "b" : "7FC244A43000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "6961067A58DEF0FA65DFFA86CC07B768F93807CE" }, { "b" : "7FC24483F000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3B0403FDFDE98D24FA2B5EA33202259FADF9E9A1" }, { "b" : "7FC244625000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1B56D55B99DF90787361ED6C2EA852562E67C486" }, { "b" : "7FC2443FE000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A8FAACEEB4CFEA6632970E1CC9E7EA333EA237EF" }, { "b" : "7FC24419C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "F5B144F9F5D9BE451C80211B34DB2CE348E039B6" } ] }}
      mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x561793079d51]
      mongod(+0x1570F69) [0x561793078f69]
      mongod(+0x157144D) [0x56179307944d]
      libpthread.so.0(+0xF5E0) [0x7fc245ba55e0]
      libc.so.6(gsignal+0x37) [0x7fc2458081f7]
      libc.so.6(abort+0x148) [0x7fc2458098e8]
      mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x561792321fab]
      mongod(+0x1284636) [0x561792d8c636]
      mongod(__wt_eventv+0x3D7) [0x56179232c2ba]
      mongod(__wt_err+0x9D) [0x56179232c4d6]
      mongod(__wt_panic+0x24) [0x56179232c72e]
      mongod(__wt_block_extlist_read+0x8F) [0x56179398100f]
      mongod(__wt_block_extlist_read_avail+0x2B) [0x56179398155b]
      mongod(__wt_block_checkpoint_load+0x26D) [0x56179397db4d]
      mongod(+0x1E7A727) [0x561793982727]
      mongod(__wt_btree_open+0xB3B) [0x56179399c32b]
      mongod(__wt_conn_btree_open+0x163) [0x5617939d5643]
      mongod(__wt_session_get_btree+0xFB) [0x561793a5fdeb]
      mongod(__wt_session_get_btree+0x63D) [0x561793a6032d]
      mongod(__wt_session_get_btree_ckpt+0x14C) [0x561793a605ec]
      mongod(__wt_curfile_open+0x161) [0x5617939e3e11]
      mongod(+0x1F4E1B8) [0x561793a561b8]
      mongod(__wt_metadata_cursor_open+0x6E) [0x561793a201fe]
      mongod(__wt_metadata_cursor+0x4B) [0x561793a202db]
      mongod(wiredtiger_open+0x1659) [0x5617939d1949]
      mongod(ZN5mongo18WiredTigerKVEngineC2ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xD20) [0x561792d70980]
      mongod(+0x12616E5) [0x561792d696e5]
      mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x697) [0x561792c5c397]
      mongod(+0x80619C) [0x56179230e19c]
      mongod(main+0x96B) [0x56179232dadb]
      libc.so.6(__libc_start_main+0xF5) [0x7fc2457f4c05]
      mongod(+0x8842B1) [0x56179238c2b1]
      ----- END BACKTRACE -----

        1. config.log
          10 kB
        2. mongo-wiredtiger.zip
          136 kB
        3. repair_attempt.tar.gz
          129 kB

            Assignee:
            daniel.hatcher@mongodb.com Danny Hatcher (Inactive)
            Reporter:
            bigfacat yuzhiyu
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: