[SERVER-36273] file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value Created: 25/Jul/18 Updated: 15/Sep/18 Resolved: 30/Jul/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.4.7 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Carlos | Assignee: | Nick Brewer |
| Resolution: | Done | Votes: | 0 |
| Labels: | envc, jail, rpo, rpu, szs, trct, wtc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | FreeBSD |
| Participants: |
| Description |
|
Hello, After an outage our db is unable to restart throwing this error: WiredTiger error (0) [1532500963:868333][16866:0x80ac06400], file:WiredTiger.wt, connection: WiredTiger.turtle: encountered an illegal file format or internal value As seen in other issues it may has to do with a corrupted WiredTiger.wt. Although our WiredTiger.turtle appears to be empty.
The complete log with more info about it: I CONTROL [main] ***** SERVER RESTARTED ***** , processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security: { authorization: "ena bled", keyFile: "/var/db/mongodb/mongodb-keyfile" }, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } } ***aborting after fassert() failure F - [initandlisten] Got signal: 6 (Abort trap). 0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c ,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"}, {"b":"1021000","o":"285803A","s":"_ZN5mongo29re portOutOfMemoryErrorAndExitEv"},{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bc }}
|
| Comments |
| Comment by Nick Brewer [ 30/Jul/18 ] |
|
carlos.lopez If you have a copy of the sizeStorer.wt file that comes from the same time that the backup.zip file was created, I would recommend using that instead. In any case, I suggest taking a look at the guidelines mentioned previously to avoid errors due to power failures in the future. Regards, |
| Comment by Carlos [ 27/Jul/18 ] |
|
We have tried the files you provided but it still throws the same error which seems to be in the sizeStorer.wt file. Our current sizeStorer.wt file was added on a later edit to the backup.zip As for your questions:
This is the new error(still looks almost the same as the old): I CONTROL [main] ***** SERVER RESTARTED ***** , processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security: { authorization: "enabled", keyFile: "/var/db/mongodb/mongodb-keyfile" }, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } } ***aborting after fassert() failure F - [initandlisten] Got signal: 6 (Abort trap). 0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c ,{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},{"b":"1021000","o":"285803A","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" } }}
|
| Comment by Nick Brewer [ 26/Jul/18 ] |
|
I've uploaded the files after a repair attempt. Could you please unpack them and substitute them for the current matching files in your dbpath? A few other things I'd like to confirm:
Thanks, |
| Comment by Carlos [ 26/Jul/18 ] |
|
We have found a backup but it appears to be corrupt in some way but at least this one has some content. This is the error it throws I CONTROL [main] ***** SERVER RESTARTED ***** , processManagement: { fork: true, pidFilePath: "/var/db/mongodb/mongod.lock" }, security: { authorization: "enabled", keyFile: "/var/db/mongodb/mongodb-keyfile" }, storage: { dbPath: "/var/db/mongodb", directoryPerDB: true, journal: { enabled: true }}, systemLog: { destination: "file", logAppend: true, path: "/var/db/mongodb/mongod.log" } } ***aborting after fassert() failure F - [initandlisten] Got signal: 6 (Abort trap). 0x3879e36 0x3879775 0x387903a 0x808bc1b4a 0x808bc122c {"backtrace":[ {"b":"1021000","o":"2858E36","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1021000","o":"2858775","s":"_ZN5mongo15clearSignalMaskEv"},{"b":"1021000","o":"285803A","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"808BB4000","o":"DB4A","s":"pthread_sigmask"},{"b":"808BB4000","o":"D22C","s":"pthread_getspecific"}],"processInfo":{ "mongodbVersion" : "3.4.7", "gitVersion" : "cf38c1b8a0a8dca4a11737581beafef4fe120bcd", "compiledModules" : [], "uname" : { "sysname" : "FreeBSD", "release" : "10.3-RELEASE-p24", "version" : "FreeBSD 10.3-RELEASE-p24 #0: Wed Nov 15 04:57:40 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC", "machine" : "amd64" }}}
We have also tried to execute it with --repair but it throws another error of a problem in a database collection wt file.
Could you please take a look?
|
| Comment by Nick Brewer [ 25/Jul/18 ] |
|
carlos.lopez It looks like the WiredTiger.turtle file you've uploaded is badly corrupted - this file contains the metadata that is used to interpret WiredTiger.wt, which is in turn used to interpret all other .wt files. With that file in its current state, any repair attempts we make are going to be unsuccessful. In this situation, our best recommendation would be to resync the affected node if it is a replica set member, or restore from a backup if one is available. Some guidelines for avoiding issues related to unreliable storage layers or server failures:
Regards, |
| Comment by Carlos [ 25/Jul/18 ] |
|
We don't have an operative backup. It's there any other option we can try? |