[SERVER-61936] WiredTiger metadata corruption detected - unable to repair Created: 07/Dec/21 Updated: 14/Feb/22 Resolved: 14/Feb/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 3.6.18 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Sarojini Jillalla | Assignee: | Edwin Zhou |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | Improper shutdown of the service causes data corruption. |
||||||||
| Participants: | |||||||||
| Description |
|
We are using Graylog, Elasticsearch and MongoDB for logging and archiving. These apps are run as docker containers with 3 replicas on 3 RHEL servers. We are using MongoDB version 3.6.18 Today, the repair is failing to recover the data. As such the MongoDB node is not able to be started properly, thereby causing the Graylog server to be down. This is the output from the repair operation. My title [root@dcvsl125 mongodb]# docker run -it -v /docker/services/mongodb/db01:/data/db mongo:3.6.18 mongod --repair Unknown macro: { bindIpAll}
, repair: true } ***aborting after fassert() failure [root@dcvsl125 mongodb]#
Attaching the files from this corrupted instance.
I have searched here and found the https://jira.mongodb.org/browse/SERVER-40088, which is almost similar to the issue I am facing currently. Can you please help me in recovering from this failure?
|
| Comments |
| Comment by Edwin Zhou [ 14/Feb/22 ] |
|
We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket. |
| Comment by Edwin Zhou [ 07/Feb/22 ] |
|
We still need additional information to diagnose the problem. If your issue is still there, would you please provide:
Best, |
| Comment by Edwin Zhou [ 28/Jan/22 ] |
|
If the issue is still there, can you please provide:
Best, |
| Comment by Sarojini Jillalla [ 07/Jan/22 ] |
|
Hi Edwin, The issue is still there and we ae not able to repair the MongoDB. Thanks & Regards Sarojini Jillalla| Sr.Consultant - DevOps|CGI Group Inc. From: Edwin Zhou (Jira) <jira@mongodb.org> EXTERNAL SENDER: Do not click any links or open any attachments unless you trust the sender and know the content is safe. https://jira.mongodb.org/s/en_USp8swtz-1988229788/6109/25/_/jira-logo-scaled.png [cid:image001.png@01D8014E.65693460] Edwin Zhou<https://urldefense.com/v3/__https:/jira.mongodb.org/secure/ViewProfile.jspa?name=edwin.zhou__;!!AaIhyw!7mIrfXeYtFI7_KWIFZr7Nz4gY1xEnzTsxHkOFe1JkNrI5T5_CqaAvI9zQ-6I7hPTxg$> mentioned you on [Bug] Re: WiredTiger metadata corruption detected - unable to repair <https://urldefense.com/v3/__https:/jira.mongodb.org/browse/SERVER-61936__;!!AaIhyw!7mIrfXeYtFI7_KWIFZr7Nz4gY1xEnzTsxHkOFe1JkNrI5T5_CqaAvI9zQ-7RQdHQIQ$> Hi Sarojini Jillalla<https://urldefense.com/v3/__https:/jira.mongodb.org/secure/ViewProfile.jspa?name=saroj.jillalla*40cgi.com__;JQ!!AaIhyw!7mIrfXeYtFI7_KWIFZr7Nz4gY1xEnzTsxHkOFe1JkNrI5T5_CqaAvI9zQ-4Jfq0l1A$>, We still need additional information to diagnose the problem. If this is still an issue for you, would you please let us know if you've found success in running --repair on the latest version on MongoDB? Best, This message was sent from MongoDB's issue tracking system. To respond to this ticket, please login to jira.mongodb.org<https://urldefense.com/v3/__https:/jira.mongodb.org__;!!AaIhyw!7mIrfXeYtFI7_KWIFZr7Nz4gY1xEnzTsxHkOFe1JkNrI5T5_CqaAvI9zQ-6y-64fDw$> using your JIRA, MongoDB Cloud Manager, or MongoDB Atlas credentials. |
| Comment by Edwin Zhou [ 05/Jan/22 ] |
|
After backing up your $dbPath, were you able to run --repair using the latest version of MongoDB? MongoDB v4.0.3 and later has improved repair functionality and may be able to resolve your corruption. If you're able to and you're still unable to launch MongoDB, please provide:
Best, |
| Comment by Sarojini Jillalla [ 04/Jan/22 ] |
|
Hi Edwin, The issue is still there and we are not able to repair the MongoDB. Please let me know what additional information is required for troubleshooting. Thanks, |
| Comment by Edwin Zhou [ 31/Dec/21 ] |
|
We still need additional information to diagnose the problem. If this is still an issue for you, would you please let us know if you've found success in running --repair on the latest version on MongoDB? Best, |
| Comment by Edwin Zhou [ 13/Dec/21 ] |
|
MongoDB 3.6 reached end of life in April of 2021. But we can provide limited guidance on this issue. The ideal resolution is to perform a clean resync from an unaffected node. First, make a complete copy of the database's $dbpath directory to safeguard so that you can work off of the current $dbpath. Then, try mongod --repair using the latest version of MongoDB. In the event that a --repair operation is unsuccessful, then please also provide:
Best, |