The objective of this ticket is to introduce a periodic cleanup job that deletes the durable state of chunks from the shard-local catalog that are unowned and no longer reachable based on the current oldest possible WT timestamp.

This tracks the correct implementation of this cleanup. There were previous attempts to do so (SERVER-122633 + SERVER-125471) that ended up being reverted.

Considerations

• The cleanup job needs to behave like a DDL, following the same schema so that it correctly serializes with other DDLs / chunk operations. We need to define the serialization mechanisms and document them.

Example

If a chunk is only visible on the current shard up to timestamp T=10, and the current oldest WT timestamp allowed for PIT reads is T=20, then we no longer need to store the chunk metadata durably and can delete it from disk (config.shard.catalog.chunks).

Why this is not strictly needed for 9.0 when releasing authoritative shards

Leftover chunks that are not reachable do not affect the correctness / durability of the cluster. There are only performance implications, and only if a shard ends up in a scenario where there is a ping-pong of chunks with different ranges. This would make the chunks collection grow, so that queries needing to refresh the filtering metadata would have to read all the on-disk data (including unowned, unreachable chunks) and then filter them out, potentially making those queries expensive. We consider this scenario rare enough that it should not block shipping authoritative shards.