[SERVER-57829] Investigation: Determine the fixed-named files which need to be deleted as part of FileBasedInitialSync Created: 17/Jun/21  Updated: 29/Oct/23  Resolved: 13/Oct/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.2.0

Type: Task Priority: Major - P3
Reporter: Matthew Russotto Assignee: Ali Mir
Resolution: Fixed Votes: 0
Labels: pm-1897-M2
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Repl 2021-10-18
Participants:

 Description   

We know WiredTiger.turtle and WiredTiger.wt need to be deleted. Check if there are any others by comparing actual directory against backup cursor. Check for both regular and encrypted storage engine, encrypted with both CBC and GSM modes.



 Comments   
Comment by Githook User [ 13/Oct/21 ]

Author:

{'name': 'ali-mir', 'email': 'ali.mir@mongodb.com', 'username': 'ali-mir'}

Message: SERVER-57829 Add fixed-name files to list of files to delete during FCBIS
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/4e609e71bb368ccdf88a2e1c36a38c58162fc688

Comment by Ali Mir [ 11/Oct/21 ]

I wrote a test to determine the file names that exist in the dbpath, but are not returned by the $backupCursor.

Here is a link to the GitHub gist. This test outputs the desired filenames at the end of execution (see log lines on 63-70). Side note, if attempting to run this locally, this test must be placed in the enterprise repo.

The output is as follows:

[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] The fixed files missing from the backup cursor output for a regular replica set: [
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"mongod.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.wt",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"diagnostic.data",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"journal",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.turtle",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"storage.bson"
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ]
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] The fixed files missing from the backup cursor output for an encrypted replica set with GCM: [
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"mongod.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"key.store",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.wt",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"diagnostic.data",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"journal",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.turtle",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"storage.bson"
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ]
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] 
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] The fixed files missing from the backup cursor output for an encrypted replica set with CBC: [
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"mongod.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"key.store",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.wt",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"diagnostic.data",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"journal",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.turtle",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"WiredTiger.lock",
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] 	"storage.bson"
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ]
[js_test:find_files_not_returned_by_backup_cursor] [jsTest] ----
[js_test:find_files_not_returned_by_backup_cursor] 

It appears that both types of encryption have the same output, and it's the same as a regular replica set with the addition of the key.store directory. Additionally, journal appears to also be a directory, and I've manually checked that the files within it are returned by the backup cursor. We may still need to manually delete the empty directory as well though.

Generated at Thu Feb 08 05:42:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.