Tiered storage currently has a local_retention option which specifies an amount of time that WT retains a local copy of files that have recently been flushed to object storage. If the system crashes during this retention period, we need to find these files and clean them up.
There are a variety of options for what this clean up might look like.
- Recovery could find these files and just delete them.
- Recovery could find these files and restart the local_retention timer for them (i.e, by queuing work items to delete them at the appropriate time). There are various permutations here – we could restart the timer at zero, or try to infer how much of the time has already elapsed.
- We could add metadata to track these retained files. For example by adding an option to the corresponding object: entry to indicate that there is a retained version of the file and when it can be removed per local_retention.
For a long term solution, I would prefer something like #3, as it would also eliminate possible race conditions around a thread trying to open the local file while another is deleting it. But it's pretty clear that local_retention is just a temporary way to get some local caching until we figure out a more general solution. So the fix here should be whatever is simplest and safest.
Whatever solution is implemented, we should add testing to make sure it is doing its job.