[SERVER-76177] Increase batch size of deletion in change collection and preimages removers Created: 17/Apr/23  Updated: 27/Apr/23  Resolved: 27/Apr/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Jordi Olivares Provencio Assignee: Jordi Olivares Provencio
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-76601 Add option to skip refetch in batched... Closed
Sprint: Execution Team 2023-05-01
Participants:

 Description   

The default batched deletes target size is currently set at 10 documents. As this is a workload that's not sensitive to latency we should maximize its throughput.

To do so we could increase the target batch size to be 32 documents. Running some workloads has shown a very significant (30-50%) improvement in time spent doing deletions.



 Comments   
Comment by Jordi Olivares Provencio [ 27/Apr/23 ]

We're cancelling this ticket in favour of SERVER-76601. That ticket will contain all the changes to batch deletes but none of the parameter tuning.

Comment by Josef Ahmad [ 19/Apr/23 ]

We can probably squeeze something further out of batch deletions, as some constraints can be relaxed for the pre-images and change collections. Some tuning to explore:

  • Disable targetBatchTimeMS (default: 5ms) in order to bundle more documents into a batch. The main goal of this parameter is to limit the amount of pinned dirty data during a batch. This is also indirectly controlled by targetStagedDocBytes, although targetBatchTimeMS is an extra guard-rail for collections many secondary indexes, which does not apply to pre-images and change collection.
  • Introduce a new BatchedDeleteStageParams entry that controls whether we should refetch a document staged for deletion if its snapshot has gone stale. Essentially, this parameter will determine whether to skip a call to ensureStillMatchesAndUpdateStats(). This refetch ensures that a document that was staged for deletion is still eligible for deletion when committing the batch. Refetching can be expensive, and becomes more likely as the batch size increases. Refetching only applies to collections with concurrent deletes or updates; this doesn't apply to pre-images and change collections.
  • All the above should help avoid most of the overhead due to refetching documents, and we should be able to increase the targetBatchDocs to a value that could be even higher than the 32 proposed in this ticket.
Comment by Jordi Olivares Provencio [ 17/Apr/23 ]

Note that doing this might increase the refetches occurring after a yield.

Generated at Thu Feb 08 06:32:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.