[SERVER-36273] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 25/Jul/18  Updated: 15/Sep/18  Resolved: 30/Jul/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.4.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Carlos Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: envc, jail, rpo, rpu, szs, trct, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive backup.zip     Zip Archive mongo.zip     File repair-attempt.tar.gz    
Operating System: FreeBSD
Participants:

 Description   

Hello,

After an outage our db is unable to restart throwing this error:

WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value

As seen in other issues it may has to do with a corrupted WiredTiger.wt. Although our WiredTiger.turtle appears to be empty.

 

The complete log with more info about it:

I CONTROL [main] ***** SERVER RESTARTED *****
I CONTROL [initandlisten] MongoDB starting : pid=16866 port=27017 dbpath=/var/db/mongodb 64-bit host=xxxxx
I CONTROL [initandlisten] db version v3.4.7
I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1s-freebsd 1 Mar 2016
I CONTROL [initandlisten] allocator: system
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/usr/local/etc/mongodb.conf", net:

{ port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security:

{ authorization: "ena bled", keyFile: "/var/db/mongodb/mongodb-keyfile" }

, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } }
W - [initandlisten] Detected unclean shutdown - /var/db/mongodb/mongod.lock is not empty.
I - [initandlisten] Detected data files in /var/db/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=64955M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=jour
nal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
E STORAGE [initandlisten] WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value
E STORAGE [initandlisten] WiredTiger error (-31804) [1532500963:868395][16866:0x80ac06400], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
I - [initandlisten]

***aborting after fassert() failure

F - [initandlisten] Got signal: 6 (Abort trap).

0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"}

,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},

{"b":"1021000","o":"285803A","s":"_ZN5mongo29re portOutOfMemoryErrorAndExitEv"}

,{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bc
d", "compiledModules" : [], "uname" :

{ "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "m achine" : "amd64" }

}}
mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x3879e36]
mongod(_ZN5mongo15clearSignalMaskEv+0x3B5) [0x3879775]
mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x9A) [0x387903a]
libthr.so.3(pthread_sigmask+0x51A) [0x808bc1b4a]
libthr.so.3(pthread_getspecific+0xE1C) [0x808bc122c]
----- END BACKTRACE -----

 



 Comments   
Comment by Nick Brewer [ 30/Jul/18 ]

carlos.lopez If you have a copy of the sizeStorer.wt file that comes from the same time that the backup.zip file was created, I would recommend using that instead. In any case, I suggest taking a look at the guidelines mentioned previously to avoid errors due to power failures in the future.

Regards,
Nick

Comment by Carlos [ 27/Jul/18 ]

We have tried the files you provided but it still throws the same error which seems to be in the sizeStorer.wt file. Our current sizeStorer.wt file was added on a later edit to the backup.zip

As for your questions:

  • What is the underlying platform for this machine (bare metal, virtual machine, container, etc)?
    • FreeBSD jail
  • What was the cause of the outage (disk corruption, power failure, etc)?
    • Power failure

 

This is the new error(still looks almost the same as the old):

I CONTROL [main] ***** SERVER RESTARTED *****
I CONTROL [initandlisten] MongoDB starting : pid=79727 port=27017 dbpath=/var/db/mongodb 64-bit host=xxx
I CONTROL [initandlisten] db version v3.4.7
I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1s-freebsd 1 Mar 2016
I CONTROL [initandlisten] allocator: system
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/usr/local/etc/mongodb.conf", net:

{ port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security: { authorization: "enabled", keyFile: "/var/db/mongodb/mongodb-keyfile" }, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } }
W - [initandlisten] Detected unclean shutdown - /var/db/mongodb/mongod.lock is not empty.
I - [initandlisten] Detected data files in /var/db/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=64955M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
E STORAGE [initandlisten] WiredTiger error (0) [1532672106:852060][79727:0x80ac06400], file:sizeStorer.wt, txn-recover: read checksum error for 4096B block at offset 278528: block header checksum of 1635017060 doesn't match expected checksum of 1479290304
E STORAGE [initandlisten] WiredTiger error (0) [1532672106:852169][79727:0x80ac06400], file:sizeStorer.wt, txn-recover: sizeStorer.wt: encountered an illegal file format or internal value
E STORAGE [initandlisten] WiredTiger error (-31804) [1532672106:852198][79727:0x80ac06400], file:sizeStorer.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
I - [initandlisten]

***aborting after fassert() failure

F - [initandlisten] Got signal: 6 (Abort trap).

0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c
----- BEGIN BACKTRACE -----

{"backtrace":[\{"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"}

,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},{"b":"1021000","o":"285803A","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" :

{ "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" }

}}
mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x3879e36]
mongod(_ZN5mongo15clearSignalMaskEv+0x3B5) [0x3879775]
mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x9A) [0x387903a]
libthr.so.3(pthread_sigmask+0x51A) [0x808bc1b4a]
libthr.so.3(pthread_getspecific+0xE1C) [0x808bc122c]
----- END BACKTRACE -----

 

Comment by Nick Brewer [ 26/Jul/18 ]

carlos.lopez

I've uploaded the files after a repair attempt. Could you please unpack them and substitute them for the current matching files in your dbpath?

A few other things I'd like to confirm:

  • What is the underlying platform for this machine (bare metal, virtual machine, container, etc)?
  • What was the cause of the outage (disk corruption, power failure, etc)?

Thanks,
Nick

repair-attempt.tar.gz

Comment by Carlos [ 26/Jul/18 ]

We have found a backup but it appears to be corrupt in some way but at least this one has some content.

This is the error it throws

I CONTROL [main] ***** SERVER RESTARTED *****
I CONTROL [initandlisten] MongoDB starting : pid=91218 port=27017 dbpath=/var/db/mongodb 64-bit host=xxx
I CONTROL [initandlisten] db version v3.4.7
I CONTROL [initandlisten] git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1s-freebsd 1 Mar 2016
I CONTROL [initandlisten] allocator: system
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/usr/local/etc/mongodb.conf", net:

{ port: 27017 }

, processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security: { authorization: "enabled", keyFile: "/var/db/mongodb/mongodb-keyfile" }, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal:

{ enabled: true }

}, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } }
W - [initandlisten] Detected unclean shutdown - /var/db/mongodb/mongod.lock is not empty.
I - [initandlisten] Detected data files in /var/db/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=64955M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
E STORAGE [initandlisten] WiredTiger error (0) [1532591534:854516][91218:0x80ac06400], file:sizeStorer.wt, txn-recover: read checksum error for 4096B block at offset 278528: block header checksum of 1635017060 doesn't match expected checksum of 1479290304
E STORAGE [initandlisten] WiredTiger error (0) [1532591534:854625][91218:0x80ac06400], file:sizeStorer.wt, txn-recover: sizeStorer.wt: encountered an illegal file format or internal value
E STORAGE [initandlisten] WiredTiger error (-31804) [1532591534:854651][91218:0x80ac06400], file:sizeStorer.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic
I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
I - [initandlisten]

***aborting after fassert() failure

F - [initandlisten] Got signal: 6 (Abort trap).

0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c
----- BEGIN BACKTRACE -----

{"backtrace":[

{"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"}

,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},{"b":"1021000","o":"285803A","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" :

{ "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" }

}}
mongod(ZN5mongo15printStackTraceERNSt3_113basic_ostreamIcNS0_11char_traitsIcEEEE+0x36) [0x3879e36]
mongod(_ZN5mongo15clearSignalMaskEv+0x3B5) [0x3879775]
mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x9A) [0x387903a]
libthr.so.3(pthread_sigmask+0x51A) [0x808bc1b4a]
libthr.so.3(pthread_getspecific+0xE1C) [0x808bc122c]
----- END BACKTRACE -----

 

We have also tried to execute it with --repair but it throws another error of a problem in a database collection wt file.

 

Could you please take a look?

backup.zip

 

 

 

Comment by Nick Brewer [ 25/Jul/18 ]

carlos.lopez It looks like the WiredTiger.turtle file you've uploaded is badly corrupted - this file contains the metadata that is used to interpret WiredTiger.wt, which is in turn used to interpret all other .wt files. With that file in its current state, any repair attempts we make are going to be unsuccessful.

In this situation, our best recommendation would be to resync the affected node if it is a replica set member, or restore from a backup if one is available.

Some guidelines for avoiding issues related to unreliable storage layers or server failures:

Regards,
Nick

Comment by Carlos [ 25/Jul/18 ]

We don't have an operative backup. It's there any other option we can try?

Generated at Thu Feb 08 04:42:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.