[SERVER-32039] Corrupt WiredTiger.wt file after crash Created: 20/Nov/17 Updated: 27/Jul/18 Resolved: 21/Nov/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.2.16 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Johannes du Plessis | Assignee: | Mark Agarunov |
| Resolution: | Done | Votes: | 0 |
| Labels: | envm, rge, rpu, trcf, wtc | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
We suffered a VM crash that housed a non replicated MongoDB server. I haven't been able to recover any of the data, using various steps found on JIRA (and StackOverflow). My best guess at this point is that the WiredTiger.wt got corrupted, so we can't start up the server. It seems VERY similar to this issue: https://jira.mongodb.org/browse/SERVER-31124?focusedCommentId=1675509&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-1675509 I'm hoping you guys can help! I've attached the MongoDB folder & also the output of the startup attempt.
|
| Comments |
| Comment by Johannes du Plessis [ 21/Nov/17 ] | |||||||||||||||||||||||||||||||||||
|
I suspected might have been the case. Thanks so much again for you help Mark - you guys have been awesome at assisting! | |||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 21/Nov/17 ] | |||||||||||||||||||||||||||||||||||
|
Hello joduplessis, Thank you for the information, I'm sorry to hear this didn't fix the issue. Unfortunately, this error indicates that there was corruption on the disk. In this situation, my best recommendation would be to resync the affected node if possible. Thanks, | |||||||||||||||||||||||||||||||||||
| Comment by Johannes du Plessis [ 21/Nov/17 ] | |||||||||||||||||||||||||||||||||||
|
Hi Mark, Okay so replaced the files - thank you for that. Tried to restart the Mongo instance, but is was failing silently this time. I checked the logs:
Regarding your other queries, we've been strung in by management to use VMs in house - with no backup availability due to limited capacity. We've been hounding them for backups, this crash was a result of a VM replication happening when it wasn't supposed to. All of your points are questionable as we don't have direct access to anything other than an SSH login of the server. Any idea what might be the cause here? | |||||||||||||||||||||||||||||||||||
| Comment by Johannes du Plessis [ 20/Nov/17 ] | |||||||||||||||||||||||||||||||||||
|
Thanks so much Mark. Will give this a try in the morning! | |||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 20/Nov/17 ] | |||||||||||||||||||||||||||||||||||
|
Hello joduplessis, Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information:
Thanks, |