Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-47102

WiredTiger.wt: encountered an illegal file format or internal value

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 4.0.0
    • Fix Version/s: None
    • Component/s: WiredTiger
    • Environment:
      Linux

      Description

      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] MongoDB starting : pid=16951 port=27017 dbpath=/app/mongodb/data/db 64-bit host=yfweb
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] db version v4.0.0
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] git version: 3b07af3d4f471ae89e8186d33bbb1d5259597d51
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] allocator: tcmalloc
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] modules: none
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] build environment:
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] distmod: rhel70
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-03-25T09:53:41.515+0000 I CONTROL [initandlisten] options: { config: "mongodb.conf", net:

      { bindIp: "0.0.0.0", port: 27017 }

      , processManagement: { fork: true }, repair: true, storage: { dbPath: "/app/mongodb/data/db" }, systemLog: { destination: "file", path: "/app/mongodb/logs/mongodb.log" } }
      2020-03-25T09:53:41.515+0000 I STORAGE [initandlisten] Detected data files in /app/mongodb/data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2020-03-25T09:53:41.515+0000 I STORAGE [initandlisten]
      2020-03-25T09:53:41.515+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
      2020-03-25T09:53:41.515+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
      2020-03-25T09:53:41.515+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3480M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
      2020-03-25T09:53:42.327+0000 E STORAGE [initandlisten] WiredTiger error (0) [1585130022:327046][16951:0x7f0a3ff2da60], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293) Raw: [1585130022:327046][16951:0x7f0a3ff2da60], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293)
      2020-03-25T09:53:42.327+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1585130022:327105][16951:0x7f0a3ff2da60], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1585130022:327105][16951:0x7f0a3ff2da60], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2020-03-25T09:53:42.327+0000 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403
      2020-03-25T09:53:42.327+0000 F - [initandlisten]

      ***aborting after fassert() failure

      • 2020-03-25T09:53:42.350+0000 F - [initandlisten] Got signal: 6 (Aborted).
        0x55d1f738e331 0x55d1f738d549 0x55d1f738da2d 0x7f0a3e7b35e0 0x7f0a3e44a3be 0x7f0a3e44b805 0x55d1f5a04c89 0x55d1f5af66a6 0x55d1f5b645b1 0x55d1f598faa1 0x55d1f598fdca 0x55d1f5b2a163 0x55d1f5b28664 0x55d1f5b094aa 0x55d1f5b62e2d 0x55d1f5b6340a 0x55d1f5b6367c 0x55d1f5bd11f6 0x55d1f5b5e9f8 0x55d1f5b27bae 0x55d1f5b27c8b 0x55d1f5b080e2 0x55d1f5ad541c 0x55d1f5ab9374 0x55d1f60e46eb 0x55d1f59889c5 0x55d1f5a6e604 0x55d1f5a06819 0x7f0a3e4370c0 0x55d1f5a6d0af

              • BEGIN BACKTRACE ----- {"backtrace":[\{"b":"55D1F500C000","o":"2382331","s":"_ZN5mongo15printStackTraceERSo"}

                ,{"b":"55D1F500C000","o":"2381549"},{"b":"55D1F500C000","o":"2381A2D"},{"b":"7F0A3E7A4000","o":"F5E0"},{"b":"7F0A3E418000","o":"323BE","s":"gsignal"},{"b":"7F0A3E418000","o":"33805","s":"abort"},{"b":"55D1F500C000","o":"9F8C89","s":"ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55D1F500C000","o":"AEA6A6"},{"b":"55D1F500C000","o":"B585B1"},{"b":"55D1F500C000","o":"983AA1","s":"wt_err"},{"b":"55D1F500C000","o":"983DCA","s":"wt_panic"},{"b":"55D1F500C000","o":"B1E163","s":"wt_turtle_read"},{"b":"55D1F500C000","o":"B1C664","s":"wt_metadata_search"},{"b":"55D1F500C000","o":"AFD4AA","s":"wt_conn_dhandle_open"},{"b":"55D1F500C000","o":"B56E2D","s":"wt_session_get_dhandle"},{"b":"55D1F500C000","o":"B5740A","s":"wt_session_get_dhandle"},{"b":"55D1F500C000","o":"B5767C","s":"wt_session_get_btree_ckpt"},{"b":"55D1F500C000","o":"BC51F6","s":"wt_curfile_open"},{"b":"55D1F500C000","o":"B529F8"},{"b":"55D1F500C000","o":"B1BBAE","s":"wt_metadata_cursor_open"},{"b":"55D1F500C000","o":"B1BC8B","s":"wt_metadata_cursor"},{"b":"55D1F500C000","o":"AFC0E2","s":"wiredtiger_open"},{"b":"55D1F500C000","o":"AC941C","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55D1F500C000","o":"AAD374"},{"b":"55D1F500C000","o":"10D86EB","s":"_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE"},{"b":"55D1F500C000","o":"97C9C5"},{"b":"55D1F500C000","o":"A62604","s":"_ZN5mongo11mongoDbMainEiPPcS1"},{"b":"55D1F500C000","o":"9FA819","s":"main"},{"b":"7F0A3E418000","o":"1F0C0","s":"__libc_start_main"},{"b":"55D1F500C000","o":"A610AF"}],"processInfo":{ "mongodbVersion" : "4.0.0", "gitVersion" : "3b07af3d4f471ae89e8186d33bbb1d5259597d51", "compiledModules" : [], "uname" :

                { "sysname" : "Linux", "release" : "4.4.180-1.el6.elrepo.x86_64", "version" : "#1 SMP Thu May 16 16:37:14 EDT 2019", "machine" : "x86_64" }

                , "somap" : [ { "b" : "55D1F500C000", "elfType" : 3, "buildId" : "596DECA05DB9B54F6E765AEC1A8AADE4665E8946" }, { "b" : "7FFEAA597000", "elfType" : 3, "buildId" : "471A8DF32A9F1908B1974DA8DAFB158FCBBDFC76" }, { "b" : "7ED4766E1000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "A38B9CE8AEAF277CBD8BC1298B1731E2C9A66192" }, { "b" : "7F0A3F8CA000", "path" : "/opt/glibc-2.14/lib/libresolv.so.2", "elfType" : 3, "buildId" : "FA7431250D27BA688CA5B69076901C13077000F8" }, { "b" : "7ED46D8E5000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "CBDA444A7109874C5350AE9CEEF3F82F749B347F" }, { "b" : "7ED467A79000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "BECFB85A8BC084042D5BF2BA9E66325CE798B659" }, { "b" : "7F0A3F075000", "path" : "/opt/glibc-2.14/lib/libdl.so.2", "elfType" : 3, "buildId" : "D69A532E8B08784D52163137CA68F79000EEFAD8" }, { "b" : "7F0A3EE5A000", "path" : "/opt/glibc-2.14/lib/librt.so.1", "elfType" : 3, "buildId" : "01EB41696F517055116AB32AD62AE124C740AE42" }, { "b" : "7F0A3EBD7000", "path" : "/opt/glibc-2.14/lib/libm.so.6", "elfType" : 3, "buildId" : "5E3C92CA474CED089DD266B56412919B2A50F4AC" }, { "b" : "7ED4715C1000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7F0A3E7A4000", "path" : "/opt/glibc-2.14/lib/libpthread.so.0", "elfType" : 3, "buildId" : "201FCBE70E81EB9B5B28CDBA5BB89346C72E8C19" }, { "b" : "7F0A3E418000", "path" : "/opt/glibc-2.14/lib/libc.so.6", "elfType" : 3, "buildId" : "17E07E976790E5BCC6CCB06F1123D140B7F2A06C" }, { "b" : "7ED479536000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "97AF4B77212F74CFF72B6C013E6AA2D74A97EF60" }, { "b" : "7ED4755E6000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "417A6ED67D904CD799091B56BDFADA0D594BB094" }, { "b" : "7ED463F95000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "1E1F213BB990982FA49BCDD2CBBFB2DB680019AC" }, { "b" : "7ED467551000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "9A737F8BF10FC99C37CC404D3FC188F6E11FEDD9" }, { "b" : "7ED467A6A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8D3D6E28DF6EB3752642A7031AAC17D39EA4265D" }, { "b" : "7ED467C3E000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "CC89B4C8CDCCD32BA610BC72784DC3B7E9BD9E19" }, { "b" : "7ED469E3A000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "7EC54D6E88BB7D2C1284117C2A483496A01EAAF4" }, { "b" : "7ED475824000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "5FA8E5038EC04A774AF72A9BB62DC86E1049C4D6" }, { "b" : "7ED465DD1000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "7B31DCDBC6EA53E83B165E972066324EB11F3674" }, { "b" : "7ED4653A4000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "B0496DD0482E729BC33F5B19CFA52787389477A0" }, { "b" : "7ED467C5C000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "B786AFA781039748E951FE1045E932BCF31900F0" }, { "b" : "7ED468A2D000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "06AD65B988B6F761D5B334BF6AEE9CFED2D98AFF" }, { "b" : "7ED468029000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "CF853ECBE1D643A626D65701261A12ABE0828BF0" }, { "b" : "7ED468224000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "810CDF602A65C7AF9CC7EA989BAFA4702B7738BF" }, { "b" : "7ED4687E5000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "36DBF3933E1ABF4DD774F7B456B5F406B632AA2A" }, { "b" : "7ED4729BD000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "8E1B3E6D72631F40825E105431067352A8565B94" }, { "b" : "7ED4627AE000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "B250057251FC715C5CCA27489506D7BBB1C9137C" }, { "b" : "7ED463194000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "90F78E5E1AF40EE481268BEAF15F9A2E3E8EFBD2" }, { "b" : "7ED465789000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0C522C589F775C324330BE09CE67DC83950A213" }, { "b" : "7ED466D86000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "AF374BAFB7F5B139A0B431D3F06D82014AFF3251" }, { "b" : "7F0A3B74E000", "path" : "/opt/glibc-2.14/lib/libcrypt.so.1", "elfType" : 3, "buildId" : "DEDC46423407162FCAC0F4C24751237E07326854" }, { "b" : "7ED472D2F000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" } ] }}
                mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55d1f738e331]
                mongod(+0x2381549) [0x55d1f738d549]
                mongod(+0x2381A2D) [0x55d1f738da2d]
                libpthread.so.0(+0xF5E0) [0x7f0a3e7b35e0]
                libc.so.6(gsignal+0x3E) [0x7f0a3e44a3be]
                libc.so.6(abort+0x175) [0x7f0a3e44b805]
                mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55d1f5a04c89]
                mongod(+0xAEA6A6) [0x55d1f5af66a6]
                mongod(+0xB585B1) [0x55d1f5b645b1]
                mongod(__wt_err+0x9D) [0x55d1f598faa1]
                mongod(__wt_panic+0x33) [0x55d1f598fdca]
                mongod(__wt_turtle_read+0x343) [0x55d1f5b2a163]
                mongod(__wt_metadata_search+0x1C4) [0x55d1f5b28664]
                mongod(__wt_conn_dhandle_open+0x8A) [0x55d1f5b094aa]
                mongod(__wt_session_get_dhandle+0xED) [0x55d1f5b62e2d]
                mongod(__wt_session_get_dhandle+0x6CA) [0x55d1f5b6340a]
                mongod(__wt_session_get_btree_ckpt+0x14C) [0x55d1f5b6367c]
                mongod(__wt_curfile_open+0x56) [0x55d1f5bd11f6]
                mongod(+0xB529F8) [0x55d1f5b5e9f8]
                mongod(__wt_metadata_cursor_open+0x6E) [0x55d1f5b27bae]
                mongod(__wt_metadata_cursor+0x4B) [0x55d1f5b27c8b]
                mongod(wiredtiger_open+0x1A12) [0x55d1f5b080e2]
                mongod(ZN5mongo18WiredTigerKVEngineC1ERKNSt7_cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0x8FC) [0x55d1f5ad541c]
                mongod(+0xAAD374) [0x55d1f5ab9374]
                mongod(_ZN5mongo23initializeStorageEngineEPNS_14ServiceContextENS_22StorageEngineInitFlagsE+0xBEB) [0x55d1f60e46eb]
                mongod(+0x97C9C5) [0x55d1f59889c5]
                mongod(ZN5mongo11mongoDbMainEiPPcS1+0x874) [0x55d1f5a6e604]
                mongod(main+0x9) [0x55d1f5a06819]
                libc.so.6(__libc_start_main+0x100) [0x7f0a3e4370c0]
                mongod(+0xA610AF) [0x55d1f5a6d0af]

              • END BACKTRACE -----

        Attachments

        1. repair_attempt.tar.gz
          14 kB
        2. wiredTigerFiles.zip
          16 kB

          Issue Links

            Activity

              People

              Assignee:
              carl.champain Carl Champain (Inactive)
              Reporter:
              1278827329@qq.com zhen wang
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: