[SERVER-38912] Repair fails WT_CURSOR.next: read checksum error Created: 09/Jan/19  Updated: 10/Jan/19  Resolved: 10/Jan/19

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: 4.0.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rémi AUGUSTE Assignee: Kelsey Schubert
Resolution: Done Votes: 0
Labels: wt-repair-success
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File WiredTiger.turtle     File WiredTiger.wt     File repair.log    
Operating System: ALL
Steps To Reproduce:

Maybe having a mongodb running on a NFS shared folder, kill it, and run two instance trying to repair.

Participants:

 Description   

I have a micro-service cluster with a NFS share, normally only one instance of mongodb (docker's mongo) is running. After a crash of the cluster I cannot bring the DB back up, the repair fails violently:

E STORAGE [initandlisten] WiredTiger error (0) [1547049030:493990][10:0x7f75f72aea00], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 28672B block at offset 16384: block header checksum of 806183382 doesn't match expected checksum of 619640434 Raw: [1547049030:493990][10:0x7f75f72aea00], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 28672B block at offset 16384: block header checksum of 806183382 doesn't match expected checksum of 619640434

 

and displays multiple chunks of data.

I'd appreciate any help in restoring the DB, it wasn't backed up and wasn't part of a replicate neither... I attached the faulty WiredTiger.wt, the mongod --repair output and the WiredTiger.turtle (as it seems to be useful for repairing).



 Comments   
Comment by Kelsey Schubert [ 10/Jan/19 ]

Thanks for the follow up, foloex. I'm glad you were able to take advantage of the improvements we made in SERVER-19815.

To prevent this type of problem in the future please take note of the following guidelines to help mitigate any issues related to unreliable storage layers or server failures.

Thank you,
Kelsey

Comment by Rémi AUGUSTE [ 09/Jan/19 ]

I managed to repair the DB using the latest mongodb (4.0.5) and it was able to run back with docker's  version (4.0.1). It's running fine now

Generated at Thu Feb 08 04:50:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.