Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29023

file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.11
    • Component/s: WiredTiger
    • Labels:
    • Linux

      Hello,
      my mongod sudden shutdown , I am unable to restart Mongod.

      This is my restart log

      2017-04-30T05:56:11.884+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] MongoDB starting : pid=61275 port=27017 dbpath=/var/lib/mongo 64-bit host=download
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] db version v3.2.11
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] git version: 009580ad490190ba33d1c6253ebd8d91808923e4
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] allocator: tcmalloc
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] modules: none
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] build environment:
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     distmod: rhel70
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     distarch: x86_64
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten]     target_arch: x86_64
      2017-04-30T05:56:11.896+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
      2017-04-30T05:56:11.932+0200 I -        [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      2017-04-30T05:56:11.932+0200 W -        [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
      2017-04-30T05:56:11.932+0200 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
      2017-04-30T05:56:11.932+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=3G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      2017-04-30T05:56:11.937+0200 E STORAGE  [initandlisten] WiredTiger (0) [1493524571:937885][61275:0x7f9caa384dc0], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
      2017-04-30T05:56:11.937+0200 E STORAGE  [initandlisten] WiredTiger (-31804) [1493524571:937937][61275:0x7f9caa384dc0], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      2017-04-30T05:56:11.937+0200 I -        [initandlisten] Fatal Assertion 28558
      2017-04-30T05:56:11.937+0200 I -        [initandlisten]
      
      ***aborting after fassert() failure
      
      
      2017-04-30T05:56:11.956+0200 F -        [initandlisten] Got signal: 6 (Aborted).
      
       0x132fa32 0x132eb89 0x132f392 0x7f9ca8ffc370 0x7f9ca8c611d7 0x7f9ca8c628c8 0x12b58b2 0x10af973 0x97194c 0x971a21 0x971be5 0x1a64c46 0x1a63696 0x1a18921 0x1aa1f7e 0x1aa2625 0x1aa2aab 0x1a27586 0x1a9d3cd 0x1a62cff 0x1a62e0e 0x1a14e63 0x1096e0f 0x10930c3 0xfbbd98 0x9b9cde 0x9bc250 0x97251d 0x7f9ca8c4db35 0x9b6607
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"F2FA32","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F2EB89"},{"b":"400000","o":"F2F392"},{"b":"7F9CA8FED000","o":"F370"},{"b":"7F9CA8C2C000","o":"351D7","s":"gsignal"},{"b":"7F9CA8C2C000","o":"368C8","s":"abort"},{"b":"400000","o":"EB58B2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"CAF973"},{"b":"400000","o":"57194C","s":"__wt_eventv"},{"b":"400000","o":"571A21","s":"__wt_err"},{"b":"400000","o":"571BE5","s":"__wt_panic"},{"b":"400000","o":"1664C46","s":"__wt_turtle_read"},{"b":"400000","o":"1663696","s":"__wt_metadata_search"},{"b":"400000","o":"1618921","s":"__wt_conn_btree_open"},{"b":"400000","o":"16A1F7E","s":"__wt_session_get_btree"},{"b":"400000","o":"16A2625","s":"__wt_session_get_btree"},{"b":"400000","o":"16A2AAB","s":"__wt_session_get_btree_ckpt"},{"b":"400000","o":"1627586","s":"__wt_curfile_open"},{"b":"400000","o":"169D3CD"},{"b":"400000","o":"1662CFF","s":"__wt_metadata_cursor_open"},{"b":"400000","o":"1662E0E","s":"__wt_metadata_cursor"},{"b":"400000","o":"1614E63","s":"wiredtiger_open"},{"b":"400000","o":"C96E0F","s":"_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb"},{"b":"400000","o":"C930C3"},{"b":"400000","o":"BBBD98","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"400000","o":"5B9CDE"},{"b":"400000","o":"5BC250","s":"_ZN5mongo13initAndListenEi"},{"b":"400000","o":"57251D","s":"main"},{"b":"7F9CA8C2C000","o":"21B35","s":"__libc_start_main"},{"b":"400000","o":"5B6607"}],"processInfo":{ "mongodbVersion" : "3.2.11", "gitVersion" : "009580ad490190ba33d1c6253ebd8d91808923e4", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-514.2.2.el7.x86_64", "version" : "#1 SMP Tue Dec 6 23:06:41 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "88BEAACF5E9DF7E467AF167F9F4F31D20015315F" }, { "b" : "7FFDC2AC2000", "elfType" : 3, "buildId" : "183CE4B56A9471419F233CCEF078E0504837ABF5" }, { "b" : "7F9CA9F17000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D0018CA5E24522ED0DC1844556FA8DBC4B39D5C3" }, { "b" : "7F9CA9B2D000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8756D2315BF50F8610875B1AFF128198FB9D202D" }, { "b" : "7F9CA9925000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77ADE22BC9FFF8D3458BD37331E7EDF174C28" }, { "b" : "7F9CA9721000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7F9CA941F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7F9CA9209000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7F9CA8FED000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2" }, { "b" : "7F9CA8C2C000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8B2C421716985B927AA0CAF2A05D0B1F452367F7" }, { "b" : "7F9CAA185000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "8F3E366E2DB73C330A3791DEAE31AE9579099B44" }, { "b" : "7F9CA89DE000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "A2499C359AA179EE23324ED949C0E508E4434F10" }, { "b" : "7F9CA86F7000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "E09A34D9083DC6FEAF7018C09D55631DEEE2836D" }, { "b" : "7F9CA84F3000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "BF54B7C8932E450769FBBB8B18864D1DD70BBC67" }, { "b" : "7F9CA82C1000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "BF8F00D7CB849ADB0B7A4703BC7B8D66AEE6A49C" }, { "b" : "7F9CA80AB000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F9CA7E9C000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "1E7A92FDD6FB3871DA97F4BCA2E147E72B6B6E1F" }, { "b" : "7F9CA7C98000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F9CA7A7E000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FE7AE845A123A3DFC0FDC2408BCBC2BA8B61B158" }, { "b" : "7F9CA7857000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "76687CA31A406854DF3BCF8D03055656F56E6892" }, { "b" : "7F9CA75F6000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "AE64AA461A26E01F60408013D361749D56DD0AE1" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x132fa32]
       mongod(+0xF2EB89) [0x132eb89]
       mongod(+0xF2F392) [0x132f392]
       libpthread.so.0(+0xF370) [0x7f9ca8ffc370]
       libc.so.6(gsignal+0x37) [0x7f9ca8c611d7]
       libc.so.6(abort+0x148) [0x7f9ca8c628c8]
       mongod(_ZN5mongo13fassertFailedEi+0x82) [0x12b58b2]
       mongod(+0xCAF973) [0x10af973]
       mongod(__wt_eventv+0x388) [0x97194c]
       mongod(__wt_err+0x8D) [0x971a21]
       mongod(__wt_panic+0x24) [0x971be5]
       mongod(__wt_turtle_read+0x2D6) [0x1a64c46]
       mongod(__wt_metadata_search+0x1F6) [0x1a63696]
       mongod(__wt_conn_btree_open+0x61) [0x1a18921]
       mongod(__wt_session_get_btree+0xDE) [0x1aa1f7e]
       mongod(__wt_session_get_btree+0x785) [0x1aa2625]
       mongod(__wt_session_get_btree_ckpt+0xAB) [0x1aa2aab]
       mongod(__wt_curfile_open+0x86) [0x1a27586]
       mongod(+0x169D3CD) [0x1a9d3cd]
       mongod(__wt_metadata_cursor_open+0x5F) [0x1a62cff]
       mongod(__wt_metadata_cursor+0x7E) [0x1a62e0e]
       mongod(wiredtiger_open+0x14B3) [0x1a14e63]
       mongod(_ZN5mongo18WiredTigerKVEngineC2ERKSsS2_S2_mbbb+0x77F) [0x1096e0f]
       mongod(+0xC930C3) [0x10930c3]
       mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x598) [0xfbbd98]
       mongod(+0x5B9CDE) [0x9b9cde]
       mongod(_ZN5mongo13initAndListenEi+0x10) [0x9bc250]
       mongod(main+0x15D) [0x97251d]
       libc.so.6(__libc_start_main+0xF5) [0x7f9ca8c4db35]
       mongod(+0x5B6607) [0x9b6607]
      -----  END BACKTRACE  -----
      

      I've uploaded my WiredTiger.* files in the hopes that someone can respond with fixed WiredTiger.* files. I would be extremely grateful...thank you!

      If anyone has any further questions, feel free to ask. Thank you!!

        1. repair_attempt.tar.gz
          3 kB
        2. wiredTiger.zip
          4 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            jonnyda jonny da
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: