[SERVER-49542] MongoDB Repair is failing -- need help Created: 16/Jul/20  Updated: 22/Jun/22  Resolved: 28/Jul/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.18
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Kalyan Kumar A Assignee: Dmitry Agranat
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Participants:

 Description   

Issue  : User has dropped collections, we have only file system level snapshot and restored to a directory. (we can see all files and everything is located in this directory only)

MongoDB Version - 4.0.18

Command Issued to repair : 

mongod --dbpath filesystempath --repair --wiredTigerCacheSizeGB 10 -vvvvvv

Errors

2020-07-16T18:58:42.090+0530 F STORAGE [initandlisten] Record store did not exist. Collection: CollectionName.XXXX.LA.1.0-00209-STD.INT-1 UUID: 2fed7adb-4c2d-4d59-8375-d64d84e36cae
2020-07-16T18:58:42.090+0530 F - [initandlisten] Fatal Assertion 50936 at src/mongo/db/catalog/database_impl.cpp 228
2020-07-16T18:58:42.090+0530 F - [initandlisten]

 

Please let me know what I am missing, as per logs I see it is fetching metadata and looks good,

 



 Comments   
Comment by Dmitry Agranat [ 28/Jul/20 ]

Thank you for the update calyankumar@gmail.com, I will go ahead and close this case.

Comment by Kalyan Kumar A [ 28/Jul/20 ]

Hi Dmitry

I have downloaded latest version and recovery is fine and able to bring
database online

Thanks

Comment by Dmitry Agranat [ 28/Jul/20 ]

Hi calyankumar@gmail.com,

If this is still an issue for you, would you please attach copies of the wiredTiger.wt and wiredTiger.turtle files?

Thanks,
Dima

Comment by Dmitry Agranat [ 21/Jul/20 ]

Hi calyankumar@gmail.com,

Please attach copies of the wiredTiger.wt and wiredTiger.turtle files and we can attempt a metadata-only repair effort using internal tools.

Keep in mind that this repair effort may not be successful, and that diagnosing corruption issues requires significant information and effort.

Thanks,
Dima

Comment by Kalyan Kumar A [ 16/Jul/20 ]

Thanks for looking into it, We have identified there is issue with one of the collection file has checksum issue.  We would like to know how to skip the corrupted collections and proceed with repair

I am sorry, I may not be able to upload complete log file because it contains some sensitive information.

wt tool log output:

 

/wt -v -h /prj/mongobackups/drgbackup/drgprd -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage /tmp/collection-200-3081784708227248719.wt
[1594914717:952843][23113:0x7f636363f740], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 16384B block at offset 3317760: block header checksum of 712246094 doesn't match expected checksum of 2763696572
[1594914717:952892][23113:0x7f636363f740], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value
[1594914717:952907][23113:0x7f636363f740], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
[1594914717:952939][23113:0x7f636363f740], txn-recover: Recovery failed: WT_PANIC: WiredTiger library panic
lt-wt: WT_PANIC: WiredTiger library panic
[root@host wiredtiger-2.7.0]#

 

 

Comment by Louis Williams [ 16/Jul/20 ]

calyankumar@gmail.com, can you please provide the full repair output?

Generated at Thu Feb 08 05:20:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.