Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-36273

file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Done
    • 3.4.7
    • None
    • WiredTiger
    • FreeBSD

    Description

      Hello,

      After an outage our db is unable to restart throwing this error:

      WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

      As seen in other issues it may has to do with a corrupted WiredTiger.wt. Although our WiredTiger.turtle appears to be empty.

       

      The complete log with more info about it:

      I CONTROL [main] ***** SERVER RESTARTED *****
      I CONTROL [initandlisten] MongoDB starting : pid=16866 port=27017 dbpath=/var/db/mongodb 64-bit host=xxxxx
      I CONTROL [initandlisten] db version v3.4.7
      I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
      I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1s-freebsd 1 Mar 2016
      I CONTROL [initandlisten] allocator: system
      I CONTROL [initandlisten] modules: none
      I CONTROL [initandlisten] build environment:
      I CONTROL [initandlisten] distarch: x86_64
      I CONTROL [initandlisten] target_arch: x86_64
      I CONTROL [initandlisten] options: { config: "/usr/local/etc/mongodb.conf", net:

      { port: 27017 }

      , processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security:

      { authorization: "ena bled", keyFile: "/var/db/mongodb/mongodb-keyfile" }

      , storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal:

      { enabled: true }

      }, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } }
      W - [initandlisten] Detected unclean shutdown - /var/db/mongodb/mongod.lock is not empty.
      I - [initandlisten] Detected data files in /var/db/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
      W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
      I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=64955M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=jour
      nal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
      E STORAGE [initandlisten] WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
      E STORAGE [initandlisten] WiredTiger error (-31804) [1532500963:868395][16866:0x80ac06400], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
      I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
      I - [initandlisten]

      ***aborting after fassert() failure

      F - [initandlisten] Got signal: 6 (Abort trap).

      0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c
      ----- BEGIN BACKTRACE -----

      {"backtrace":[\{"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"}

      ,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},

      {"b":"1021000","o":"285803A","s":"_ZN5mongo29re portOutOfMemoryErrorAndExitEv"}

      ,{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bc
      d", "compiledModules" : [], "uname" :

      { "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "m achine" : "amd64" }

      }}
      mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x3879e36]
      mongod(_ZN5mongo15clearSignalMaskEv+0x3B5) [0x3879775]
      mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x9A) [0x387903a]
      libthr.so.3(pthread_sigmask+0x51A) [0x808bc1b4a]
      libthr.so.3(pthread_getspecific+0xE1C) [0x808bc122c]
      ----- END BACKTRACE -----

       

      Attachments

        1. backup.zip
          2.04 MB
        2. mongo.zip
          2.02 MB
        3. repair-attempt.tar.gz
          1.97 MB

        Activity

          People

            nick.brewer Nick Brewer
            carlos.lopez Carlos
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: