[SERVER-63204] Delete the temp db path on the recipient replica set once the migration is complete. Created: 02/Feb/22  Updated: 19/Aug/23  Resolved: 03/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Suganthi Mani Assignee: [DO NOT USE] Backlog - Server Serverless (Inactive)
Resolution: Duplicate Votes: 0
Labels: shard-merge-milestone-3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-61144 Finish importing donated collections ... Closed
Duplicate
duplicates SERVER-72215 Delete the donor WT files that are un... Closed
Related
related to SERVER-74614 Complete TODO listed in SERVER-63204 Closed
related to SERVER-80262 Complete TODO listed in SERVER-63204 Closed
Assigned Teams:
Serverless
Sprint: Server Serverless 2022-02-21
Participants:

 Description   

Make sure we delete the temp dir at the correct time / place in the code on migration commit or abort. This is something we consider doing asynchronously through the TenantFileImporterService as it might slow things down to do it synchronously in the OpObserver.

When state machine is GC'ed. Detect the GC in an op observer.



 Comments   
Comment by A. Jesse Jiryu Davis [ 07/Mar/22 ]

Random thought for consideration: on rollback, restart, or state change, always delete all directories in dbpath whose names match the pattern migrationTmpFiles.migrationId. That will ensure we don't miss any edge cases and slowly fill the disk.

Comment by Christopher Caplinger [ 17/Feb/22 ]

sending this back to the backlog until we figure out who will handle this cleanup

Comment by A. Jesse Jiryu Davis [ 10/Feb/22 ]

That's a good idea, especially since migration abort could be related to a recipient restart or rollback. In those scenarios it's hard for all recipient nodes to reliably remember that they must delete the temp directory. I'll ask Cloud in the design if they have a preference.

Comment by Suganthi Mani [ 10/Feb/22 ]

jesse Currently, in shard Merge Milestone-2 design, we say

As in MT Migrations, Cloud is responsible for deleting imported D databases from R if the migration aborts.

If that's the case, can we also make them to delete the temp DB path on migration abort? And, we delete the db path only on R migration state transitions to kCopiedFiles, via TenantFileImporterService?

Comment by A. Jesse Jiryu Davis [ 10/Feb/22 ]

The copied files can be large, so it's useful to clean them up ASAP, rather than waiting minutes for GC. This avoids out-of-disk in tests and real life where we do multiple merges into the same recipient in rapid succession.

Comment by Christopher Caplinger [ 10/Feb/22 ]

jesse suganthi.mani is there any reason we wouldn't want to just handle tmp dir deletion via the op observer when state machine garbage collection happens? or is there a reason we need to perform this cleanup in two places? The description makes it sound like we should clean this up for the abort case via the op observer and in some other place on migration commit.

Comment by Suganthi Mani [ 02/Feb/22 ]

CC jesse

Generated at Thu Feb 08 05:57:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.