[SERVER-63204] Delete the temp db path on the recipient replica set once the migration is complete. Created: 02/Feb/22 Updated: 19/Aug/23 Resolved: 03/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Suganthi Mani | Assignee: | [DO NOT USE] Backlog - Server Serverless (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | shard-merge-milestone-3 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Serverless
|
||||||||||||||||||||||||||||
| Sprint: | Server Serverless 2022-02-21 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
Make sure we delete the temp dir at the correct time / place in the code on migration commit or abort. This is something we consider doing asynchronously through the TenantFileImporterService as it might slow things down to do it synchronously in the OpObserver. When state machine is GC'ed. Detect the GC in an op observer. |
| Comments |
| Comment by A. Jesse Jiryu Davis [ 07/Mar/22 ] |
|
Random thought for consideration: on rollback, restart, or state change, always delete all directories in dbpath whose names match the pattern migrationTmpFiles.migrationId. That will ensure we don't miss any edge cases and slowly fill the disk. |
| Comment by Christopher Caplinger [ 17/Feb/22 ] |
|
sending this back to the backlog until we figure out who will handle this cleanup |
| Comment by A. Jesse Jiryu Davis [ 10/Feb/22 ] |
|
That's a good idea, especially since migration abort could be related to a recipient restart or rollback. In those scenarios it's hard for all recipient nodes to reliably remember that they must delete the temp directory. I'll ask Cloud in the design if they have a preference. |
| Comment by Suganthi Mani [ 10/Feb/22 ] |
|
jesse Currently, in shard Merge Milestone-2 design, we say
If that's the case, can we also make them to delete the temp DB path on migration abort? And, we delete the db path only on R migration state transitions to kCopiedFiles, via TenantFileImporterService? |
| Comment by A. Jesse Jiryu Davis [ 10/Feb/22 ] |
|
The copied files can be large, so it's useful to clean them up ASAP, rather than waiting minutes for GC. This avoids out-of-disk in tests and real life where we do multiple merges into the same recipient in rapid succession. |
| Comment by Christopher Caplinger [ 10/Feb/22 ] |
|
jesse suganthi.mani is there any reason we wouldn't want to just handle tmp dir deletion via the op observer when state machine garbage collection happens? or is there a reason we need to perform this cleanup in two places? The description makes it sound like we should clean this up for the abort case via the op observer and in some other place on migration commit. |
| Comment by Suganthi Mani [ 02/Feb/22 ] |
|
CC jesse |