[SERVER-36339] Wiredtiger corrupted after unclean shutdown Created: 28/Jul/18  Updated: 04/Sep/18  Resolved: 01/Aug/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 3.6.2
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: KaiJun Assignee: Nick Brewer
Resolution: Done Votes: 0
Labels: envm, rpo, rpu, trcf, wtc
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS Linux release 7.4.1708


Attachments: File WiredTiger.turtle     File WiredTiger.wt     HTML File dmesg     Text File mongod.log     Text File mongod_updated.log     File repair-attempt.tar.gz    
Operating System: Linux
Participants:

 Description   

Hi, I saw some previous bug report, i believe i ran into the same issue. Mongo is not able to be repaired. Log attached along with WiredTiger.wt

Cheers,

KaiJun



 Comments   
Comment by KaiJun [ 01/Aug/18 ]

Ok, got it. Thank you so much for the help.

KaiJun

Comment by Nick Brewer [ 01/Aug/18 ]

kjchew82 

Thanks for the additional information. Unfortunately, this error indicates that there was corruption on the disk, most often caused by a faulty storage layer. In this situation, our best recommendation would be to [resync the affected node|https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/] or [restore from a backup|https://docs.mongodb.com/manual/core/backups/] if possible.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures. 

-Nick

Comment by KaiJun [ 30/Jul/18 ]

Hi Nick, it is always 3.6.2.

Cheers,
KaiJun

Comment by Nick Brewer [ 30/Jul/18 ]

kjchew82 Has this machine always been running on 3.6.2, or was it upgraded recently?

Thanks,
Nick

Comment by KaiJun [ 30/Jul/18 ]

Hi Nick,

Replaced, new error regarding journal appear, updated log attached.

Environment: vm instances from cloud provider with SAN storage.
Cause of crash: networking hardware migrations, causing SAN disconnected.

dmesg attached.

Cheers,
KaiJun

mongod_updated.log dmesg

Comment by Nick Brewer [ 30/Jul/18 ]

kjchew82 I've attached the files after a repair attempt. Would you please extract these files, substitute them for the current ones in your $dbpath, and let us know if it resolves the issue?

Additionally, I'd like to confirm the following:

  • The underlying cause of the crash (power failure, disk corruption, etc)
  • The environment (native, virtualized, container)

Thanks,
Nick

repair-attempt.tar.gz

Generated at Thu Feb 08 04:42:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.