[SERVER-34392] file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value Created: 09/Apr/18 Updated: 14/Aug/18 Resolved: 02/May/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.19 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Frank Perez | Assignee: | Kelsey Schubert |
| Resolution: | Done | Votes: | 0 |
| Labels: | docker, envc, rpo, rps, trcf, wtc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
Hi, We had a major power outage and it knocked down our mongo db running in docker as part of a stack. Now it won't come back up. Mongo had it's own volume mounted to /data/db. At first I took the stack down created and ran a temporary container mounted to the same volume to see if I could repair the data. I ran the "mongod --repair --dbpath /data/db" and nothing, It was still saying "file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1843462872 doesn't match expected checksum of 1205067277". I also ran this command "mongod --repair --dbpath /data/db --storageEngine wiredTiger"...same thing. I then went as far as to restore the data from a backup that we had done the day prior of the volume used to store the mongo data. When I deployed the stack again (fresh containers, etc.) and I check the logs that the server is outputting...still the same issue. Please advise/help as I thought that restoring from the backup if anything would of fixed it but still nothing. I'm attaching the stdout of the repair command as well as the regular log of the server while running (or trying to anyways). I just don't understand why the backup restore isn't working as the backup was done before the outage therefor it shouldn't be corrupted. Please help me recover the db as it's very important. Much appreciation and thanks. |
| Comments |
| Comment by Frank Perez [ 02/May/18 ] |
|
Thank you Kelsey for the update! |
| Comment by Kelsey Schubert [ 02/May/18 ] |
|
HI frankp, We have improvements in this space currently scheduled, see Additionally we suggest the following to help mitigate any issues related to unreliable storage layers or server failures:
Kind regards, |
| Comment by Frank Perez [ 10/Apr/18 ] |
|
Kelsey, Thank you very much for the repaired files, it worked!!! Thanks again for the help and any light you can shed on the questions above would be greatly appreciated. Sincerely, |
| Comment by Kelsey Schubert [ 09/Apr/18 ] |
|
Hi frankp, I've attached a repair attempt, repair_attempt.tar.gz Thank you, |
| Comment by Frank Perez [ 09/Apr/18 ] |
|
files have been uploaded. |
| Comment by Frank Perez [ 09/Apr/18 ] |
|
ok. will do. thanks |
| Comment by Kelsey Schubert [ 09/Apr/18 ] |
|
Hi frankp, Would you please upload the wiredtiger.wt and wiredtiger.turtle files so we can attempt a repair? Thank you, |
| Comment by Frank Perez [ 09/Apr/18 ] |
|
P.S I'm running the mongo:3.2 docker image. |