Hello,

After an outage our db is unable to restart throwing this error:

WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

As seen in other issues it may has to do with a corrupted WiredTiger.wt. Although our WiredTiger.turtle appears to be empty.

 

The complete log with more info about it:

I CONTROL [main] ***** SERVER RESTARTED *****
I CONTROL [initandlisten] MongoDB starting : pid=16866 port=27017 dbpath=/var/db/mongodb 64-bit host=xxxxx
I CONTROL [initandlisten] db version v3.4.7
I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1s-freebsd 1 Mar 2016
I CONTROL [initandlisten] allocator: system
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/usr/local/etc/mongodb.conf", net:

{ port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security:

{ authorization: "ena bled", keyFile: "/var/db/mongodb/mongodb-keyfile" }

, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } }
W - [initandlisten] Detected unclean shutdown - /var/db/mongodb/mongod.lock is not empty.
I - [initandlisten] Detected data files in /var/db/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=64955M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=jour
nal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
E STORAGE [initandlisten] WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
E STORAGE [initandlisten] WiredTiger error (-31804) [1532500963:868395][16866:0x80ac06400], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
I - [initandlisten]

***aborting after fassert() failure

F - [initandlisten] Got signal: 6 (Abort trap).

0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"}

,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},

{"b":"1021000","o":"285803A","s":"_ZN5mongo29re portOutOfMemoryErrorAndExitEv"}

,{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bc
d", "compiledModules" : [], "uname" :

{ "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "m achine" : "amd64" }

}}
mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x3879e36]
mongod(_ZN5mongo15clearSignalMaskEv+0x3B5) [0x3879775]
mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x9A) [0x387903a]
libthr.so.3(pthread_sigmask+0x51A) [0x808bc1b4a]
libthr.so.3(pthread_getspecific+0xE1C) [0x808bc122c]
----- END BACKTRACE -----